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The  rqxjrt  addresses  our  evaluation  of  the  effectiveness  of  the  Defense  Acquisition 
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DEFENSE  ACQUISITION  BOARD  REVIEW  PROCESS 
EXECUTIVE  SUMMARY 


Introduction.  This  report  is  our  sixth  in  a  series  of  summary  reports  issued  since  1984 
that  evaluated  the  Defense  Acquisition  Board  (DAB)  review  process.  The  DoD 
Components  submit  documentation  for  DAB  reviews  highlighting  the  status  of  the 
program  and  its  readiness  to  proceed  into  the  next  phase  of  Ihe  acquisition  cycle  and 
make  recommendations  on  cost-schedule-performance  trade-offs  for  decision  by  the 
Under  Secretary  of  Defense  for  Acquisition  and  Technology.  The  DAB  review  should 
provide  an  independent  assessment  of  the  program  and  verify  that  exit  criteria  and  the 
minimum  required  accomplishments  of  the  phase  preceding  the  milestone  have  been 
completed. 

Objectives.  The  audit  objective  was  to  evaluate  the  effectiveness  of  the  DAB  review 
process,  including  the  adequacy  of  documentation  provided  to  the  DAB  in  support  of 
major  milestone  and  program  reviews.  Internal  controls  related  to  the  objective  were 
also  assessed. 

Audit  Results.  Implementation  of  the  Februpy  1991  revisions  to  the  DoD  5000  series 
of  acquisition  regulations  has  resulted  in  significant  improvement  in  the  quality  of 
documentation  used  in  the  DAB  process.  Overall,  DAB  documentation  was  well- 
prepared  and  met  requirements.  However,  we  identified  a  systemic  deficiency  in  cost 
and  operational  effectiveness  analyses.  The  Military  Dq^ments  did  not  always 
perform  adequate  cost  and  operational  effectiveness  analyses.  Additionally,  user 
involvement  in  the  cost  and  operational  effectiveness  analysis  process  could  be 
improved.  Consequently,  program  alternatives  and  the  impact  of  acquisition  program 
baseline  breaches  were  not  properly  considered  in  making  program  decisions. 

Internal  Controls.  The  audit  identified  a  material  internal  control  weakness;  controls 
did  not  always  ensure  that  cost  and  operational  effectiveness  analyses  were  performed 
when  needed.  Part  I  of  the  rq)ort  discusses  this  internal  control  weakness. 

Potential  Benefits  of  Audit.  Implementation  of  the  recommendations  in  this  report 
will  enhance  the  effectiveness  of  the  DAB  review  process.  Three  of  the  nine  audit 
rqK>rts  on  major  Defense  acquisition  programs  being  summarized  in  this  rq)ort 
identified  potential  monetary  benefits.  However,  no  potential  monetary  benefits  are 
associated  with  the  finding  and  recommendations  in  this  summary  rqx)rt  on  the  DAB 
review  process  (Appendix  F). 

Summary  of  Recommendations.  We  recommended  more  timely  identification  and 
approval  of  requirements  for  cost  and  operational  effectiveness  analyses. 

Management  Comments.  The  Director,  Acquisition  Program  Integration,  provided 
management  comments  to  the  draft  report  for  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology.  The  Director  indicated  general  concurrence  with  our 
recommendations  for  the  finding  on  cost  and  operation^  effectiveness  analyses.  He 
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nonconcurred  with  the  recommendation  that  cost  and  operational  effectiveness  analyses 
be  reviewed  and  updated,  if  needed,  when  a  breach  in  acquisition  program  baseline 
occurred.  He  partially  concurred  with  the  recommendations  that  the  Joint 
Requirements  Oversight  Council  assess  the  alternatives  to  use  in  cost  and  operational 
effectiveness  analyses  and  the  adequacy  of  the  analyses  performed.  In  commenting  on 
the  draft  report  fmding  on  exit  criteria,  the  Director  stated  that  our  recommendations 
would  add  levels  of  complexity  to  the  bureaucracy  that  the  Administration  was 
attempting  to  reduce.  Part  n  contains  a  discussion  of  management  comments  to  the 
report.  Part  IV  contains  the  complete  text  of  management  comments. 

Audit  Response.  We  deleted  the  draft  finding  on  the  use  of  exit  criteria  in  view  of 
management  comments  that  recommended  changes  would  not  be  made  without  more 
conclusive  examples  of  weaknesses  in  the  current  use  of  exit  criteria.  We  plan  to 
initiate  a  separate  audit  of  this  subject  if  it  is  still  warranted  when  current  actions  to 
streamline  the  acquisition  review  process  are  completed.  In  response  to  this  final 
report,  we  request  that  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology 
reconsider  the  position  taken  concerning  the  recommendation  to  review  and  update  tte 
cost  and  operational  effectiveness  analyses,  if  needed,  when  a  breach  in  acquisition 
program  baseline  occurs.  We  also  request  an  estimated  date  for  revision  of  DoD 
Instruction  5000.2  and  the  results  of  coordination  with  the  Joint  Requirements 
Oversight  Council  be  provided  in  the  reply  to  this  final  rqwrt  by  February  28,  1995. 
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Part  I  -  Introduction 


Background 


This  report  is  the  sixth  in  a  series  of  summary  reports  prepared  since  1984  that 
evaluate  the  Defense  Acquisition  Board  (DAB)  process,  llie  DAB  is  the  senior 
advisory  b^y  to  the  Under  Secreh^  of  Defense  for  Acquisition  and 
Technology^  (USD[A&T])  responsible  for  enforcing  policies  and  procedures 
governing  the  operations  of  the  DoD  acquisition  process.  The  acquisition 
process  is  structured  in  distinct  logical  phases  separated  by  major  decision 
points,  called  milestones.  DAB  reviews  are  held  at  each  major  milestone 
decision  point  in  the  acquisition  process.  Each  decision  point  le^s  to  a  phase 
of  the  acquisition  process  (Appradk  A). 

When  the  USD(A&T)  feels  that  issues  for  acquisition  programs  should  be 
resolved  between  milestone  reviews,  a  DAB  program  review  addresses  those 
special  issues. 

The  DAB  is  the  primary  forum  to  advise  the  USD(A&T)  on  mission  needs 
approved  by  the  Joint  Requirements  Oversight  Council  for  Milestone  0  reviews, 
possible  concept  exploration  and  definition  study  efforts.  Milestone  I  through  IV 
decision  point  reviews,  and  program  reviews  of  major  Defense  acquisition 
programs  subject  to  DAB  oversight.  The  reviews  ensure  that  a  program  is 
ready  to  proceed  into  more  advanced  stages  of  development  or  prc^uction 
before  receiving  USD(A&T)  approval  and  that  proposed  program  plans  for 
subsequent  stages  are  consistent  with  sound  acquisition  management  practices. 

The  DAB  is  supported  by  three  committees:  the  Strategic  Systems  Committee; 
the  Conventional  Systems  Committee;  and  Command,  Control, 
Communications,  and  the  Intelligence  Systems  Committee.  The  cognizant  DAB 
Committee  has  several  responsibilities.  Each  Committee  conducts  planning 
meetings  to  establish  program  document  requirements  and  documentation 
reviews  to  ensure  that  aU  exit  criteria  and  the  minimum  required 
accomplishments  of  the  phase  preceding  the  milestone  are  complete  and  to 
provide  the  basis  for  the  preparation  of  the  integrated  program  assessment  of  the 
program  for  presentation  to  the  DAB.  The  DAB  Committees  identify  the  issues 
to  be  addressed  at  the  DAB  review  and  make  key  recommendations  for 
USD(A&T)  action. 

After  the  Committee  review,  the  DAB  conducts  the  milestone  review  meeting 
focusing  on  four  questions  pertinent  to  approving  a  program  to  proceed  to  the 
next  acquisition  phase: 

o  Where  is  the  program  (versus  where  it  should  be)? 

o  Where  is  the  program  going  (and  how  wiU  it  get  there)? 

o  What  risks  exist  (and  how  will  those  risks  be  managed)? 


1  Formerly  the  Under  Secretary  of  Defense  for  Acquisition  (USD[A]). 
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o  Is  the  plan  for  the  program  affordable? 

After  a  full  discussion  at  the  DAB  review  of  the  issues,  trade-offs,  baselines, 
exit  criteria,  and  proposed  acquisition  strategy,  the  USD(A&T)  determines  the 
actions  to  be  taken.  USD(A&T)  decisions  resulting  from  DAB  reviews  are  in 
an  acquisition  decision  memorandum,  signed  by  the  USD(A&T). 


Objectives 


The  overall  audit  objective  was  to  evaluate  the  effectiveness  of  the  DAB  review 
process.  Specifically,  we  examined  the  adequacy  of  documentation  provided  by 
the  Military  Departments  and  Defense  agencies  to  the  Defense  Acquisition 
Board  in  support  of  major  milestones  and  program  reviews.  Further,  we 
evaluated  compliance  with  DoD  Directive  StXW.l,  "Defense  Acquisition," 
February  23,  1^1;  DoD  Instruction  5000.2,  "Defense  Acquisition  Management 
Policies  and  Procedures,"  February  23,  1991;  DoD  Manual  5000.2-M, 
"Defense  Acquisition  Management  Documentation  and  Reports,"  February  23, 
1991;  and  congressional  direction  issued  to  the  Office  of  die  Secreta^  of 
Defense  (OSD).  We  also  reviewed  applicable  internal  controls. 


Scope  and  Methodology 


We  performed  this  program  audit  in  accordance  with  auditing  standards  issued 
by  the  Comptroller  General  of  the  United  States,  as  implemented  by  the 
Inspector  General,  DoD,  and  accordingly  included  such  tests  of  internal  controls 
as  were  deemed  necessary.  The  audit  was  conducted  from  July  1992  through 
December  1993  and  reviewed  data  and  information  dated  from  1975  through 
1993  relating  to  four  major  Defense  acquisition  programs  discuss^  in  diis 
report.  We  have  also  consolidated  the  findings  and  recommendations 
concerning  the  three  major  Defense  acquisition  programs  reviewed  in  the  prior 
phase  of  this  audit  into  &s  report  because  of  the  direct  correlation  of  die  issues 
identified. 

The  acquisition  programs  selected  for  the  FYs  1992  and  1993  audits  of  the  DAB 
review  process  and  the  order  in  which  they  were  reviewed  were  the  Sensor 
Fuzed  Weapon  (SFW),  F/A-18  E/F  Aircraft,  M1A2  ABRAMS  Tank,  Airborne 
Self-Protection  Jammer  (ASPJ),  AU  Source  Analysis  System  (AS AS), 
Multifunctional  Information  Distribution  System  (^DS),  and  DDG-51  Aegis 
Destroyer.  This  report  also  summarizes  information  from  audits  of  the  C-17 
Aircraft  and  Short  Range  Attack  Missile  n  (SRAM  II)  programs  that  included 
findings  involving  the  DAB  review  process.  Although  the  President  terminated 
the  SRAMn  Program  on  Sq)tember27,  1991,  in  a  nuclear  arms  reduction 
initiative,  it  is  in  this  DAB  summary  report  because  our  review  of  the  SRAM  n 
Program  indicated  systemic  problems  with  the  DAB  review  process. 
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We  performed  audit  field  work  related  to  the  DAB  review  process  at  OSD  and 
Military  Department  organizations  responsible  for  the  preparation  and  review  of 
DAB-required  documents.  We  also  visited  selected  contractor  facilities.  A  list 
of  organizations  visited  or  contacted  is  in  Appendix  G. 

We  assessed  the  reliability  of  the  computer-processed  data  used  as  audit 
evidence  to  the  extent  necessary  to  accompUsh  the  audit  objectives.  Data  testing 
was  performed  to  confirm  the  completeness  and  authenticity  of  evidence  used  to 
support  the  findings  and  recommendations  related  to  the  nine  major  Defense 
acquisition  programs  reviewed. 


Internal  Controls 


We  assessed  the  implementation  of  requirements  in  DoD  Directive  5010.38, 
"DoD  Internal  Management  Control  Program,"  April  14,  1987,  and  the 
internal  management  controls  used  to  conduct  Defense  Acquisition  Board 
reviews  and  prepare  supporting  DAB  documentation.  Those  controls  and 
procedures  are  sj^ified  in  DoD  Directive  5000.1,  "Defense  Acquisition,"  and 
DoD  Instruction  5000.2,  "Defense  Acquisition  Management  Policies  and 
Procedures,"  February  23,  1991.  The  audit  identified  material  internal  control 
weaknesses,  as  defined  DoD  Directive  5010.38.  The  controls  were  not 
effective  related  to  cost  and  operational  effectiveness  analyses  (COEAs)  and 
status  reporting  against  approved  exit  criteria.  Neither  area  was  identified  as  an 
assessable  unit  within  the  established  internal  management  control  program.  As 
a  result,  vulnerability  assessments  and  management  control  reviews  were  not 
performed  for  those  areas.  We  intend  to  perform  an  audit  specifically 
examining  the  implementation  of  the  internal  management  control  program  for 
major  Defense  acquisition  programs.  Therefore,  separate  recommendations 
addressing  the  internal  management  control  program  are  not  in  this  report. 
Copies  of  the  final  report  will  be  provided  to  the  senior  officials  responsible  for 
internal  controls  within  OSD  and  Uie  Military  Departments. 


Prior  Audits  and  Other  Reviews 


Since  1984,  we  have  issued  five  summary  audit  reports  evaluating  the  DAB  and 
the  Defense  Systems  Acquisition  Review  Council  (the  predecessor  to  die  DAB) 
processes.  Ilie  results  of  those  summary  reports  are  briefly  presented  in 
Appendix  B.  This  summary  report  is  based  on  our  review  of  nine  major 
programs,  addressed  in  10  audit  reports  issued  between  December  1991  and 
December  1993  with  DAB-related  findings.  The  results  of  these  10  reports, 
including  findings,  recommendations,  and  management  comments,  are 
synopsiz^  in  Appendix  C. 
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Another  Matter  of  Interest 


The  Office  of  the  Inspwtor  General,  DoD,  issued  Report  No.  94-014,  "Low- 
Rate  Initial  Production  in  Major  Defense  Acquisition  Programs,"  November  9, 
1993,  recommending  that  entry  into  low-rate  initial  production  requires  a 
milestone  review  equivalent  in  stature  to  the  four  milestone  reviews  in  the 
present  DAB  process.  In  response  to  our  recommendation,  the  Office  of  the 
USp(A&T)  indicated  that,  as  an  alternative  corrective  action,  it  would  be 
willing  to  implement  a  more  rigorous  review  at  Milestone  n  that  would 
determine  whether  the  next  milestone  should  more  appropriately  be  a  Milestone 
in,  full-rate  production  approval,  or  a  Milestone  HIA,  low-rate  production 
approval.  However,  implementation  of  this  alternative  corrective  action  is 
currently  m  abeyance  pending  the  results  of  a  specially  formed  Process  Action 
Team  review  to  streamline  the  entire  Defense  acquisition  process.  Certain 
deficiencies  noted  in  this  audit  would  potentially  have  been  avoided  by  a  more 
thorough  review  of  acquisition  programs  before  entering  low-rate  initial 
production. 
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Part  n  -  Finding  and  Reconunendations 


Cost  and  Operational  Effectiveness 
Analysis 

The  Military  Departments  did  not  always  perform  adequate  cost  and 
operational  effectiveness  analyses  (COEA)  to  determine  the  cost  and 
operational  advantages  and  disadvantages  of  program  alternatives, 
including  the  impact  of  changes  in  assumptions  such  as  performance 
characteristics  and  cost.  COEAs  were  not  performed  because  DoD 
guidance  did  not  provide  adequate  lead  times  between  establishing 
COEA  documentation  requirements  and  the  scheduled  DAB  review  and 
because  COEA  updates  were  not  mandatory  when  a  program  breached 
its  acquisition  program  baseline.  Deficiencies  in  the  COEAs  occurred 
because  the  alternatives  to  be  assessed  and  measures  of  effectiveness  to 
be  used  were  not  approved  at  the  start  of  the  analysis  by  the  milestone 
decision  authority  with  input  from  the  Joint  Requirements  Oversight 
Council.  Additionally,  the  Joint  Requirements  Oversight  Council  did 
not  provide  an  assessment  of  the  COEA  results  to  the  Defense 
acquisition  executive  in  support  of  the  DAB  milestone  review.  As  a 
result,  programmatic  decisions  were  being  made,  including  continuation 
of  the  current  acquisition  phase  and  entry  to  the  next  phase,  without  fully 
considering  the  cost-effectiveness  analyses  of  various  program 
alternatives  or  the  impact  of  breaches  of  acquisition  program  basetines. 
In  addition,  programs  were  delayed  to  accomplish  analyses  where 
requirements  were  identified  late  in  the  DAB  review  process. 


Background 


DoD  Instruction  5000.2,  "Defense  Acquisition  Management  Policies  and 
Procedures,"  February  23,  1991,  part  4,  and  DoD  Manual  5000.2-M,  "Defense 
Acquisition  Management  Documentation  and  Reports,"  February  23,  1991, 
part  8,  provide  procedures  and  guidelines  for  COEAs.  The  COEAs  are  to 
evaluate  the  costs  and  benefits,  such  as  operational  effectiveness  or  military 
utility,  of  alternative  actions  to  meet  recognized  Defense  needs.  The  COEAs 
are  required  to  be  prepared  and  considered  at  milestone  decision  reviews.  They 
aid  decisionmaking,  facilitate  communications,  and  document  acquisition 
decisions  by  highlighting  the  advantages  and  disadvantages  of  the  alternatives 
being  consider^  and  showing  the  sensitivity  of  each  ^temative  to  possible 
changes  in  key  assumptions,  such  as  threat,  or  variables,  including  selected 
performance  capabilities.  Further,  COEAs  provide  early  identification  and 
discussion  of  reasonable  alternatives  among  decisionmakers  and  staffs  at  all 
levels.  Disagreements  on  key  assumptions  and  variables  must  be  explicitly 
identified  and  should  not  be  immers^  in  the  presentation  of  a  compromise 
X)sition.  The  COEA  provides  the  rationale  for  the  decisions  by  providing  a 
tistorical  record  of  alternatives  at  each  milestone.  Additionally,  COEAs  must 
lave  thresholds  that  are  the  maximum  cost  or  minimum  accqitable  performance 
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Aat  can  be  tolerated  in  a  system.  The  COEA  draws  upon  system 
interrelationships  because  individual  systems  generally  cannot  be  evaluated  in 
isolation  and  few  deficiencies  can  be  resolved  by  just  one  system. 

The  DoD  Component  responsible  for  the  mission  area  in  which  a  deficiency  or 
opportunity  is  identified  normally  prepares  the  COEA.  The  DoD  Component 
he^  identifies  the  ind(^ndent  analysis  organization  to  prepare  the  COEA.  TTie 
Joint  Staff  re^nsibility  is  to  ensure  that  the  full  range  of  alternatives  is 
considered,  organizational  and  operational  plans  are  developed,  and  Joint- 
Service  issues  are  addressed. 

The  Director,  Program  Analysis  and  Evaluation  (D[PA&E])^,  has  primary 
responsibility  for  assessing  the  adequacy  of  COEAs  submitt^  in  support  of 
DAB  reviews.  The  D^A&E)  provides,  as  necessary,  guidance  tailor^  to  the 
program  under  review  in  the  DAB  review  procedures  from  the  USD(A&T).  In 
the  DAB  process,  the  COEA  is  required  at  milestone  decision  reviews  starting 
with  Milestone  I.  At  Milestones  HI  and  IV,  the  analysis  updates  the  previous 
analysis  required  at  Milestone  n.  The  milestone  decision  authority  specifies  the 
elements  of  the  updated  analysis  for  a  Milestone  HI  review  as  part  of  the 
pre-milestone  planning  process. 


Cost  and  Operational  Effectiveness  Evaluation 


COEAs  were  not  always  performed  when  required  or  did  not  always  adequately 
address  viable  alternatives  to  proceeding  with  new  major  Defense  acquisition 
programs.  The  Military  Departments  did  not  perform  COEAs  to  determine  the 
advantages  and  disadvantages  of  alternatives  or  the  impact  on  cost  and 
operational  effectiveness  of  changes  in  key  program  assumptions  for  six  of  the 
nine  individual  weapon  systems  reviewed:  F/A-18  E/F  aircraft,  M1A2  tank, 
C-17  aircraft,  SRAM  II  missile,  ASPJ  jammer,  or  DDG-51  destroyer. 

The  DoD  policy  and  procedures  related  to  the  content,  performance,  and 
reporting  of  the  results  of  COEAs  required  improvement.  Specifically,  the 
alternatives  to  be  considered  in  a  COEA  and  the  measures  of  effectiveness  used 
to  compare  the  alternatives  were  not  specifically  approved  by  the  milestone 
decision  authority  before  the  start  of  the  COEA.  As  a  result,  the  analyses 
actually  performed  were  limited  in  wme  cases,  such  as  the  F/A-18  E/F,  C-17, 
and  pDG-51  programs,  to  alternatives  that  excluded  viable  solutions  to  the 
mission  need.  The  milestone  decision  authority,  in  conjunction  with  the  Joint 
Staff,  has  the  implied  responsibility  to  ensure  that  the  full  range  of  reasonable 
alternatives  is  considered.  However,  DoD  Instruction  5000.2  does  not  specify 
how  eiAer  the  decision  authorify  or  the  Joint  Staff  will  fulfill  this  critical 
responsibility  or  the  timing  of  the  approval  of  alternatives  to  be  considered 
related  to  the  scheduled  DAB  milestone  or  program  review.  Because  DoD 
Manual  5000.2M  states  that  determining  the  alternatives  to  be  used  in  the 


2  Formerly  the  Assistant  Secretary  of  Defense  (Program  Analysis  and  Evaluation). 
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COE  A  is  one  of  the  most  important  stq)s  in  developing  the  analysis,  the  absence 
of  spwific  procedures  for  review  and  approval  of  the  COEA  alternatives  is  a 
material  we^ess  in  DoD  acquisition  policy. 

The  DoD  Instruction  5000.2,  part  4,  section  E,  on  COEAs  states  that  the 
D(PA&E)  will  provide,  as  necessary,  guidance  t^ored  to  the  program  under 
review  and  that  r^resentatives  of  that  omce  and  the  cognizant  DAB  Committee 
should  participate  m  the  early  stages  of  development  of  the  COEA.  The  nature 
of  their  guidance  is  not  address^  in  the  DoD  Instruction.  This  direction  is 
insufficient  to  ensure  that  both  the  alternatives  to  be  analyzed  and  the  measures 
of  effectiveness  to  be  used  in  the  analysis  are  specifically  approved  by  the 
decision  authority  and  the  Joint  Staff  as  the  rroresentative  for  the  eventual  user. 
Additionally,  the  guidance  is  provided  at  the  time  of  the  DAB  planning  meeting, 
which  occurs  6  months  before  the  scheduled  DAB  review.  The  draft  DAB 
documentation,  which  includes  the  COEA,  must  be  provided  to  the  cognizant 
DAB  Committee  about  60  days  before  the  scheduled  DAB  review.  As  a  result, 
only  about  120  days  are  available  to  prepare  the  COEA  under  existing  policy 
and  procedures  in  DoD  Instruction  5000.2. 

In  addition,  key  measures  of  effectiveness  were  not  determined  sufficiently 
before  the  start  of  the  COEA  to  permit  validation  bv  the  D(PA&E)  and  approval 
by  the  milestone  decision  authority.  Measures  or  effectiveness  are  defined  to 
measure  operational  capabilities  in  terms  of  engagement  or  battle  outcomes  of 
program  alternatives  compared  to  a  common  baseline.  In  accordance  with  DoD 
Instruction  5000.2,  measures  of  effectiveness  should  be  developed  at  a  level  of 
specificity  such  that  a  system's  effectiveness  during  developmental  and 
operational  testing  can  be  assessed  with  the  same  effectiveness  criteria  as  used  in 
the  COEA.  This  level  of  specificity  permits  further  refinement  of  the  COEA  to 
reassess  cost-effectiveness  compared  to  alternatives  if  that  performance,  as 
determined  during  testing,  indicates  a  significant  drop  in  effectiveness  compared 
to  levels  assumed  in  the  COEA.  Approval  of  the  alternatives  to  be  considered 
in  the  COEA  and  the  measures  of  effectiveness  to  be  used  in  the  analysis  would 
help  ensure  that  the  resulting  analysis  was  sufficient  to  support  the  appropriate 
acquisition  decision.  Further,  the  milestone  decision  authority  did  not  approve 
the  independent  organizations  designated  to  perform  COEAs. 

DoD  Instruction  5000.2,  part  13,  section  D,  "Joint  Requirements  Oversight 
Council  Review  Procedures,"  discusses  Joint  Requirements  Oversight  Council 
responsibilities  for  review  of  mission  need  statements,  operational  requirements 
documents,  and  acquisition  program  baselines  in  support  of  the  overall  DAB 
review  process.  However,  the  DoD  Instruction  does  not  address  eiAer  Joint 
Requirements  Oversight  Council  review  of  the  measures  of  effectiveness  before 
perfonnance  of  the  COEA  or  the  results  of  the  COEA  as  a  basis  for  the  Joint 
Requirements  Oversight  Council  assessment  provided  to  the  DAB.  As  discussed 
in  DoD  Instruction  5000.2,  i^4,  section  E,  the  COEA  depends  upon 
supporting  andyses  of  the  mission  need,  threat  and  U.S.  capabilities,  system 
interrelationships,  and  multi-role  systems.  A  description  of  those  analyses 
extracted  from  DoD  Instruction  5000.2  is  in  Appendix  D.  While  review  or  the 
mission  need  statements,  operational  r^uirements  documents,  and  acquisition 
program  baselines  by  the  Joint  Requirements  Oversight  Council  is  essential, 
approval  of  those  documents  does  not  automatically  approve  the  supporting 
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analyses  to  the  COEA  because  of  significant  differences  in  the  content  of  the 
documents  and  because  of  the  introduction  of  assumptions  in  the  COEA 
pertaining  to  such  factors  as  system  employment  and  enemy  tactics.  As  shown 
in  Appendix  D,  the  COEA  depends  heavily  on  the  input  of  the  user,  yet  the 
process  for  providing  the  input  can  clearly  be  improve^!.  Therefore,  the  Joint 
Requirements  Oversight  Council  should  be  r^uired  to  assess  the  COEA  and  the 
DAB  documents  already  required  to  be  reviewed  as  a  basis  for  providing  the 
Joint  Requirements  Oversight  Council  assessment  to  the  DAB. 

For  example,  the  mission  need  analysis  supporting  the  COEA  assesses  the 
strengths  and  weaknesses  of  a  military  force  confronting  a  postulated  threat  in  a 
specified  scenario  or  set  of  circumstances  such  as  force  structure,  geographic 
location,  and  environmental  conditions.  Consistent  with  die  Defense  Planning 
Guidance,  scenarios  are  selected  and  the  COEA  assesses  how  the  alternatives 
under  consideration  would  contribute  to  the  national  military  mission. 
However,  those  scenarios  are  not  defined  in  detail  in  the  DAB  documents 
reviewed  by  the  Joint  Requirements  Oversight  Council,  and  the  Joint 
Requirements  Oversight  Council  is  not  required  to  provide  specific  input  to  the 
D(PA&E)  on  the  mission  need  analysis  or  other  elements  of  the  COEA,  such  as 
the  assumptions  made,  that  can  significantly  impact  the  results  of  the  analysis. 


Cause  for  the  Lack  of  a  COEA 


COEAs  were  not  performed  because  DoD  guidance  did  not  provide  adequate 
lead  times  between  establishing  DAB  documentation  ^uirements  and 
scheduling  the  actual  DAB  review.  The  DoD  guidance  establishes  a  process  for 
Defense  acquisition  executive  and  Joint  Requirements  Oversight  Council 
approval  of  the  critical  and  underlying  elements  of  the  COEA  to  be  performed. 

L«ad  Time.  DoD  Instruction  5000.2,  part  13,  discusses  DAB  review 
procedures  and  states  that  the  DAB  milestone  review  process  will  begin  with  a 
planning  meeting  at  least  6  months  before  the  DAB  milestone  review. 
One  purpose  of  the  milestone  planning  meeting  is  to  assess  the  plans  for  key 
milestone  documents  such  as  the  COEA,  indqiendent  cost  estimate,  test  and 
evaluation  master  plan,  and  acquisition  strategy  report.  The  6-month  lead  time 
between  the  DAB  planning  meeting  and  the  actual  DAB  milestone  review 
appears  to  be  insufficient,  considering  all  requirements  associated  with  a  COEA 
as  outlined  in  DoD  Manual  5(XX).2-M,  part  8.  The  lead  time  should  be  at  least 
1  year  for  assessing  the  plans  for  a  COEA.  Additionally,  to  avoid  program 
schedule  delays  and  disruption  caused  by  questioning  COEA  requirements 
immediately  preceding  DAB  reviews,  the  Defense  acquisition  executive  should 
review  and  approve  the  alternatives  and  measures  of  effectiveness  in  the  COEA 
before  its  start.  The  Joint  Requirements  Oversight  Council  and  D(PA&E) 
should  base  their  approval  on  the  COEA  requirements  recommended  by  the 
cognizant  Service  acquisition  executive.  Further,  to  safeguard  the  independence 
of  the  COEA  perform^,  the  Defense  acquisition  executive  should  also  approve 
the  indq)endent  organization  designated  to  perform  the  COEA  so  that  the  need 
for  supplemental  analysis  is  minimized  or  eliminated.  Approval  of  the 
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assumptions,  alternatives,  and  measures  of  effectiveness  used  and  the 
independent  organization  to  perform  the  COEA  is  especially  needed  because, 
under  current  procedures,  organizations  perform  repeated  cost  benefit  and 
effectiveness  analyses  that  vary  widely  in  assumptions,  measures  of 
effectiveness,  and  alternatives  assessed. 

Program  Breaches.  DoD  Instruction  5000.2  provides  that  the  milestone 
decision  authority  may  hold  a  program  review  before  approving  a  recommended 
baseline  change,  but  the  Manual  does  not  require  an  update  of  a  COEA  when  a 
program  breaches  its  acquisition  program  baseline.  While  such  updates  need 
not  be  a  complete  reaccomplishment  of  the  COEA,  the  impact  of  the  baseline 
breach  on  the  continued  cost  and  operational  effectiveness  of  the  system  should 
be  assessed  and  would  provide  the  USD(A&T)  with  an  effective  tool  for 
evaluating  the  status  of  the  program.  The  impact  of  the  breaches  for  the  C-17, 
SRAMn,  and  M1A2  Tank  Programs  should  have  resulted  in  updates  to 
respective  COEAs  to  provide  USD(A&T)  additional  data  for  evaluating  the 
programs.  The  COEA  updates  may  also  have  eliminated  or  reduced  the 
management  oversight  those  programs  experienced  as  a  result  of  the  breaches. 
DoD  Instruction  5000.2  requires  that  COEAs  include  a  sensitivity  analysis  to 
highlight  the  impact  of  changes  in  factors  such  as  threat,  key  performance 
criteria,  and  baseline  parameters.  If  breaches  in  the  acquisition  program 
baseline  exceed  the  ranges  considered  in  the  sensitivity  analysis  performed,  an 
update  of  the  COEA  should  normally  be  required. 


Effect  of  Not  Performing  COEAs 


The  cost  and  operational  effectiveness  of  various  program  alternatives  was  not 
being  fully  considered  before  proceeding  with  programmatic  decisions.  The 
objective  of  a  COEA  is  to  focus  on  fiilfUling  nussion  needs  rather  than 
supporting  continued  procurement  of  a  particular  system  or  platform.  However, 
we  found  that  the  analyses  often  were  tailored  to  support  the  recommended 
alternative.  The  COEA  is  absolutely  fundamental  to  sound  acquisition  decisions 
and  should  be  afforded  a  high  priority  in  preparation  for  DAB  reviews. 
Strengthening  the  procedures  associated  wift  timely  approval  of  COEA 
requirements  will  ensure  that  the  resulting  analysis  is  sufficient  to  support  the 
decision  to  be  made.  We  recognize  that  the  timing  of  the  start  of  tiie  COEA  can 
be  impacted  by  other  DAB  documentation,  such  as  program  cost  estimating, 
affordability  assessments,  and  test  and  evaluation  master  plans.  However, 
earlier  completion  of  those  documents  would  provide  more  time  for  preparation 
of  DAB  documentation  and  facilitate  program  management  of  major  Defense 
acquisition  programs.  Further,  the  COEA  sensitivity  analysis  can  be 
sufficiently  broad  to  encompass  all  COEA  assumptions,  and  the  models  often 
can  be  revised  to  recompute  cost-effectiveness  comparisons. 

The  increased  involvement  of  the  Joint  Requirements  Oversight  Council  in 
approving  COEA  alternatives  and  measures  of  effectiveness  before  conducting 
the  analysis  and  reviewing  the  resulting  analysis  including  tiie  validation  of  the 
assumptions  used  would  ensure  that  the  user  requirements  are  properly 
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translated  into  the  COEA.  Debate  regarding  viable  alternatives  or  measures  of 
effectiveness  should  not  be  delayed  until  scheduled  DAB  reviews  are  imminent. 
Early  user  involvement  should  ensure  the  original  COEA  is  done  properly. 

Review  and  update,  if  required,  of  the  COEA  in  support  of  revisions  to  the 
acquisition  program  baseline  can  limit  the  need  for  DAB  program  reviews  or 
other  forms  of  additional  oversight.  Sp^ifically,  if  the  COEA  is  still  valid  after 
considering  the  revisions  to  the  acquisition  program  baseline,  then  the  need  for 
oversight  reviews  to  consider  acquisition  program  baseline  revisions  should  be 
minimized. 


Recommendations,  Management  Comments,  and  Audit 
Response 


We  recommend  that  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  revise  DoD  Instruction  5000.2,  "Defense  Acquisition 
Management  Policies  and  Procedures,"  February  23,  1991,  to  include 
requirements  for: 

1.  A  cost  and  operational  effectiveness  analysis  supporting  a  Defense 
Acquisition  Board  milestone  or  program  review  be  established  at  least 
12  months  before  the  scheduled  review  and  include  specific  approval  by  the 
milestone  decision  authority  of  the  altematives  to  be  assessed,  measures  of 
effectiveness  to  be  utilized,  and  the  independent  organization  designated  to 
perform  the  analysis. 

Management  Comments.  The  Director,  Acquisition  Program  Integration, 
responded  for  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology. 
The  Director  concurred  indicating  that  DoD  Instruction  5000.2  would  be 
revised  to  require  cost  and  operational  effectiveness  analyses  to  be  started 
12  months  before  the  DAB  as  a  normal  procedure.  The  complete  text  is  in  Part 
IV. 

Audit  Response.  We  consider  the  Director's  comments  responsive.  However, 
in  response  to  the  final  report,  please  estimate  when  revision  of  DoD  Instruction 
5000.2  may  be  expected. 

2.  Review  and  update,  if  needed,  existing  cost  and  operational  effectiveness 
analyses  as  a  part  of  the  approval  process  for  baseline  revisions  when  a 
program  breaches  its  acquisition  program  baseline. 

Management  Comments.  The  Director  nonconcurred,  stating  that  given  the 
sheer  number  of  acquisition  program  baseline  breaches,  implementing  a 
recommendation  to  automatically  update  cost  and  operational  effectiveness 
analyses  when  acquisition  program  baseline  breaches  occur  would  shut  down 
their  entire  acquisition  process.  The  Director  noted  that  the  nature  and 
magnitude  of  an  acquisition  program  baseline  breach  may  not  necessitate  an 
update  of  the  cost  and  operationid  effectiveness  analysis  and  that  an  update  of 
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the  cost  and  operational  effectiveness  analysis  could  take  6  to  12  months.  The 
Director  also  felt  that  the  recommendation  would  be  a  massive  overkill  that 
would  violate  the  spirit  of  the  National  Performance  Review,  which  found  an 
overwhelming  num^r  of  complex  regulations. 

Audit  Response.  The  Director  misread  the  recommendation.  We  agree  that 
update  of  a  program's  cost  and  operational  effectiveness  analysis  may  not 
always  be  necessary.  The  recommendation  is  that  the  cost  and  operational 
effectiveness  analysis  should  be  reviewed  and  updated,  if  needed  (underscoring 
added  for  emphasis),  as  a  part  of  the  approval  process  for  baseline  revisions. 
The  USD(A&T)  generally  would  need  to  know  at  least  the  rough  magnitude  of 
the  impact  of  a  baseline  breach  on  the  continued  cost  and  operational 
effectiveness  of  a  system. 

Management  comments  indicate  that  needed  updates  of  cost  and  operational 
effectiveness  analyses  should  not  be  made  "automatically”  because  such  updates 
would  require  signiflcant  amounts  of  work.  Our  recommendation  is  ba^  on 
the  logic  that  if  the  facts  and  circumstances  upon  which  a  program's  cost  and 
operational  effectiveness  analysis  were  made  have  changed,  then  updates  are 
often,  but  not  always,  needed  to  determine  the  continued  cost  and  operational 
effectiveness  of  the  system.  Given  the  cost  of  major  Defense  acquisition 
programs,  DoD's  best  interest  is  to  get  the  information  needed  to  make  sound 
management  decisions,  even  when  significant  amounts  of  work  are  required. 

The  National  Performance  Review  cites  the  need  to  eliminate  unnecessary  and 
unproductive  rules,  but  nothing  in  this  report  should  be  construed  as  advocating 
procedures  that  are  unsound.  Based  on  this  clarification  of  the  intent  of  our 
recommendation,  we  request  that  the  Office  of  the  Under  Secretary  of  Defense 
for  Acquisition  and  Technology  reconsider  its  position  in  response  to  the  final 
report. 

3.  The  Joint  Requirements  Oversight  Council  to  provide  an  assessment  to 
the  Defense  acquisition  executive  of  the  aitematives  to  be  considered  in  cost 
and  operational  eRectiveness  analyses  on  miqor  Defense  acquisition 
programs  and  the  measures  of  effectiveness  to  he  used  before  the  start  of 
the  analysis. 

Management  Comments.  The  Director  partially  concurred,  stating  that  early 
involvement  of  the  Joint  Requirements  Oversight  Council  in  the  cost  and 
operational  effectiveness  analysis  process  would  be  useful,  but  that  the  Joint 
Requirements  Oversight  Council  had  not  responded  to  his  office  on  the 
recommendation.  The  Director  indicated  diat  the  issue  would  be  discussed  with 
the  Joint  Requirements  Oversight  Council. 

Audit  Response.  The  Director's  comments  were  partially  responsive.  In  reply 
to  the  final  rq)ort,  please  estimate  when  coordination  with  the  Joint 
Requirements  Oversight  Council  will  be  completed  and  indicate  management's 
position  on  the  recommendation. 
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4.  The  Joint  Requirements  Oversight  Council  to  provide  an  assessment  to 
the  Director,  Program  Analysis  and  Evaluation,  on  the  adequacy  of  the  cost 
and  operational  eftiectiveness  analyses  performed  on  m^or  Defense 
acquisition  programs. 

Management  Comments.  The  Director  partially  concurred,  stating  that  the 
Director  for  Program  Analysis  and  Evaluation  would  welcome  assessment  of  the 
alternatives  to  be  considered  in  cost  and  operational  effectiveness  analyses  by 
the  Joint  Requirements  Oversight  Council,  but  noted  that  the  Joint  Requirements 
Oversight  Council  had  not  responded  to  his  office  on  the  recommendation.  The 
Director  indicated  that  the  issue  would  be  discussed  with  the  Joint  Requirements 
Oversight  Council. 

Audit  Response.  We  are  pleased  with  the  comments  on  this  recommendation. 
In  response  to  the  final  report,  please  estimate  when  coordination  with  the  Joint 
Requirements  Oversight  Council  will  be  completed  and  management's  final 
position  on  the  recommendation. 

Management  Comments  on  the  Finding.  In  Appendix  E,  we  provide  audit 
responses  to  specific  management  comments  to  the  draft  report  pertaining  to  the 
content  of  the  two  draft  audit  findings. 


Management  Comments  Required 


Management  is  requested  to  comment  on  the  items  indicated  with  an  "X"  in  the 
following  table. 


Management  Comments  Required  on  Finding 


Responses  Should  Cover 


Concur 

Nonconcur 

Proposed 

.Action 

Completion 

Date 

Related 

Issues 

1. 

USD(A&T) 

X 

2. 

USD(A&T) 

X 

X 

X 

Internal 

Control 

3. 

USD(A&T) 

X 

X 

X 

4. 

USD(A&T) 

X 

X 

X 
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Appendix  A.  Summary  of  Milestone  Decision 

Points  and  Acquisition  Phases 


The  milestone  decision  process  starts  with  the  identification  of  broadly  stated 
mission  needs  that  cannot  be  satistied  by  nonmateiiel  solutions. 


Milestone  0.  Concept  Studies  Approval  marks  the  initial  formal  interface 
between  broadly  stated  requirements  and  acquisition  management  sjrstems.  For 
potential  programs  passing  this  decision  point,  studies  of  alternative  material 
concepts  are  conducted  to  identify  the  most  promising  solutions  to  valid  user 
needs. 


Milestone  I.  Concept  Demonstration  Approval  marks  the  start  of  a  new 
acquisition  program.  At  this  decision  point,  the  concept  baseline  containing 
initial  program  cost,  schedule,  and  performance  objectives  is  established. 
Programs  passing  this  decision  point  enter  the  Demonstration  and  Validation 
phase  in  which  technologies  critical  to  the  most  promising  concepts  are 
demonstrated  and  incorporated  into  system  design. 


Milestone  n.  Development  Approval  establishes  the  development  baseline 
containing  refined  program  cost,  schedule,  and  performance  objectives. 
Programs  passing  this  decision  point  enter  the  Engineering  and  Manufacturing 
Development  phase  in  which  the  most  promising  design  approaches  developed 
in  the  preceding  phase  are  translated  into  stable,  producible,  and  cost-effective 
design  systems. 


Milestone  El.  Production  Approval  establishes  a  production  baseline 
containing  refined  program  cost,  schedule,  and  performance  objectives. 
Programs  passing  this  decision  point  enter  the  Production  and  Dq)loyment 
phase  in  which  a  stable,  efficient  production  and  support  base  is  established. 
An  operational  capability  that  satisfies  the  mission  n^  must  also  be  achieved 
during  this  phase. 


Milestone  IV.  Major  Modification  Approval  determines  whether  major 
modification  of  a  system  already  in  production  is  warranted.  Programs  passing 
this  decision  point  enter  the  Operations  and  Supjwrt  phase  in  which  systems  are 
monitored  to  ensure  that  they  continue  to  provide  &e  capabilities  required  to 
meet  mission  needs  and  that  identified  shortcomings  are  corrected. 
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Acquisition  Board  Summary  Audit 
Reports 


Since  1984,  we  have  issued  five  summary  audit  reports  evaluating  the  DAB  and 
the  Defense  Systems  Acquisition  Review  Council  (the  predecessor  to  the  DAB) 
processes. 

Report  No.  91-032,  "Summary  Report  on  the  Audit  of  the  Effectiveness  of 
the  Defense  Acquisition  Board  Process— FY  1988,"  January  25,  1991.  This 
report  summarized  the  results  of  audits  covering  five  systems  but  did  not  contain 
findings  or  recommendations.  The  report  stated  that  actions  had  been  taken 
regarding  the  recommendations  in  the  reports  on  the  individual  systems.  In 
addition,  Congress  and  OSD  took  numerous  actions  subsequent  to  Uie  issuance 
of  the  system  reports. 

Report  No.  87-193,  "Summary  Report  on  the  Audit  of  the  E^ectiveness  of 
the  Defense  Systems  Acquisition  Review  Council  (DSARC)  Process— 
FY  1986,"  July  17,  1987.  This  report  summarized  Ae  results  of  audits 
covering  seven  individual  systems  but  did  not  find  new  systemic  issues  that 
would  require  policy  changes  and,  therefore,  did  not  contain  new  findings  and 
recommendations.  However,  it  did  reaffirm  prior  observations  that  more 
rigorous  enforcement  of  existing  acquisition  policy  was  needed  and  that 
recommendations  contained  in  prior  summary  reports  remained  valid  and 
unaccomplished. 

Report  No.  87-166,  "Summary  Report  on  the  Audit  of  the  Effectiveness  of 
the  Defense  Systems  Acquisition  Review  Council  (DSARC)  Process— 
Hiasein,"  June  3,  1987.  This  report  summarized  the  results  of  audits 
covering  six  systems  and  presented  two  summary  findings  based  on  systemic 
conditions  noted  during  the  review.  The  first  finding,  "Operational 
Thresholds,"  reported  that  program  managers  had  to  provide  adequate 
operational  performance  thresholds  at  the  Milestone  n  decision  point  to  serve  as 
overall  operational  objectives  to  be  met  before  receiving  production  approval 
from  the  Secretary  of  Defense.  The  second  finding,  "Major  Changes  to 
Acquisition  Programs,"  reported  that  the  Navy  and  the  Air  Force  restructured 
the  acquisition  strategies  for  two  major  programs  without  obtaining  the 
permission  of  the  Secretary  of  Defense. 

Report  No.  85-104,  "Summary  Report  on  tiie  Audit  of  the  EHectiveness  of 
the  Defense  Systems  Acquisition  Review  Council  (DSARC)  Process— 
niasen,"  July  28,  1985.  This  rqx)rt  summarized  the  results  of  audits 
covering  10  systems  and  presented  2  summary  findings  based  on  systemic 
conditions  not^  during  the  review.  The  first  finding,  "Acquisition  Strategy," 
reported  that  required  Milestone  I  reviews  were  not  held  or  planned  for  6  of  the 
10  major  systems  in  the  audit.  The  second  finding,  "Justification  for  a  Major 
System  New  Start  Document,"  reported  that  the  new  start  documents  for 
five  major  systems  did  not  adequately  present  all  necessary  information,  as 
required  by  the  5000  series  of  DoD  Directives. 
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Report  No.  85-009,  "Sununary  Report  on  the  Audit  of  the  Effectiveness  of 
the  Defense  Systems  Acquisition  Review  Council  (DSARC)  Process," 
October  22,  1984,  This  report  summarized  the  results  of  audits  covering 
six  systems  and  presented  two  summai^  findings  based  on  systemic  conditions 
noted  during  the  audit,  "^e  first  finding,  "DSARC  Documentation,"  reported 
that  requir^  documentation  was  not  always  prepared  and  provided  to  the 
DSARC  within  established  time  frames.  The  second  finding,  "Impact  of 
Funding  Ispes,"  reported  that  the  Milita:^  Departments  had  not  adequately 
identified  either  the  source  or  the  amount  or  funding  required  to  accomplish  the 
system  development  objectives  for  the  six  systems  reviewed. 
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Appendix  C.  Synopses  of  System  Audit  Reports 

of  the  Defense  Acquisition  Board 
Process 


^tween  December  1991  and  December  1993,  we  issued  10  audit  reports, 
involving  9  major  Defense  acquisition  programs,  with  DAB-related  findings. 


Report  No.  94-019,  "The  DDG-51  Destroyer  Program  as  Part  of  the  Audit 
of  the  Defense  Acquisition  Board  Review  Process~FY  1993,"  December  10, 
1993.  This  report  found  that  the  Navy  is  retiring  surface  combatants  earlier 
than  nonnally  scheduled  without  considering  the  cost-effectiveness  of  upgrades 
to  existing  ships  instead  of  new  DDG-51  procurement.  The  report  found  that 
the  Navy  had  not  performed  an  adequate  COEA  supporting  the  Milestone  IV 
DAB  review.  The  analysis  submitt^  did  not  meet  the  requirements  of  DoD 
Manual  5000.2.  Without  an  adequate  COEA,  viable  alternatives  such  as 
upgrades,  lower  cost  ship  programs,  and  alternative  mixes  of  deployed  ships 
were  not  adequately  addressed.  The  Navy  could  reduce  DDG-51  Destroyer 
procurements  by  eight  ships  and  reduce  funding  requirements  by  about 
$4.3  billion  in  FY  1993  constant  dollars  over  the  FY  1994  Future  Years 
Defense  Plan.  Additionally,  the  total  buy  of  DDG-51  Destroyers  could  also  be 
reduced  as  a  result  of  performance  of  a  COEA  on  all  alternatives.  Also,  cost 
growth  on  future  contracts  and  higher-than-estimated  inflation  rates  would 
increase  the  reduction  in  funding  requirements  projected  over  the  Five  Year 
Defense  Plan. 

We  recommended  that  procurement  of  DDG-51  Destroyers  be  limited  to 
two  ships  per  year  through  the  remainder  of  the  FY  1994  Future  Years  Defense 
Plan,  retirement  of  operabl)'  capable  surface  combatants  be  deferred,  and  a 
formal  COEA  and  affordability  assessment  of  the  DDG-51  Destroyer  Program 
to  include  the- 21st  Century  Destroyer  Program  be  completed  before  award  of 
further  DDG-51  production  contracts.  On  February  24,  1994,  the  Director, 
Acquisition  Program  Integration  responded  to  the  report.  We  determined  that 
USD(A&T)  and  Navy  had  taken  or  planned  actions  subsequent  to  the  issuance 
of  the  report  that  were  sufficient  to  resolve  all  recommendations. 


Report  No.  93-120,  "Review  of  the  Multifunctional  Information 
Distribution  System  as  a  Fart  of  the  Audit  of  the  Defense  Acquisition  Board 
Review  Process-FY  1993,"  June  21,  1993.  This  rqwrt  found  that  the  DAB 
process  had  not  been  effective  for  the  MIDS  Program.  In  the  3  years  since 
MIDS  trmsitioned  to  a  DAB-cogni^t  major  Defense  acquisition  program,  no 
DAB  review  was  held,  no  acquisition  program  baseline  was  approved,  and  no 
exit  criteria  were  established  as  prerequisites  for  entry  into  the  Engineering  and 
Manufacturing  Development  phase.  Also,  the  impact  on  interoperability  of  Air 
Force  withdrawal  from  the  Program  was  not  adequately  addressed.  The 
international  agreement  for  the  current  phase  was  concluded  without  a  DAB 
review  and  before  an  acquisition  strategy  was  approved  and  was  not  signed  by 
the  proper  authority.  Further,  not  all  documentation  required  by  regulations  for 
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the  upcoming  Milestone  11  review  was  being  prepared;  however,  the  Navy  was 
complying  fully  with  the  more  limited  document  requirements  established  by  the 
Offic^  of  the  Secretary  of  Defense  for  the  MIDS  Program  and  associated 
acquisition  regulations  concerning  document  content.  Finally,  Ae  Navy  planned 
for  the  Semce  acquisition  executive  rather  than  the  USD(A&T)  to  make  the 
low-rate  initial  production  decision. 

We^  recomrnended  that  procedures  be  developed  to  require  a  DAB  program 
review  within  6  months  of  a  decision  by  the  USD(A&’I^  to  transition  a  program 
to  a  DAB-cognizant  major  I^fense  acquisition  program.  We  also  recommended 
that^  a  DAB  program  review  be  held  for  the  MIDS  Program  during  the 
Engin^ring  and  Manufacturing  Development  phase  as  a  basis  for  the  first  low- 
rate  initial  production  decision  and  that  a  development  test  and  evaluation  rqwrt 
and  early  operational  assessment  report  be  prepared  for  the  review. 
Additionally,  we  recommended  that  the  USDCA&T^  conclude  international 
agreement  for  Engineering  and  Manufacturing  Development.  We  further 
recommended  that  the  Joint  Requirements  (^ersight  Council  perform  an 
assessment  of  Navy,  Air  Force,  and  allied  aircraft  interoperability  before  the 
upcoming  DAB  Milestone  n  review  of  MIDS  to  evaluate  the  Air  Force 
withdrawal  from  the  Program.  The  Office  of  the  USD(A&T)  partiaUy 
concurred  with  our  finding  and  recommendations. 


Report  No.  93-087,  "Review  of  the  All  Source  Analysis  System  as  a  Part  of 
the  Audit  of  the  Effectiveness  of  the  Defense  Acquisition  Board  Review 
Process~FY  1993,"  April  20,  1993.  This  report  found  that  the  DAB  process 
was  not  effective  for  the  ASAS  Program.  Testing  conducted  and  planned  was 
inadequate  to  support  a  decision  to  field  the  Block  I  system  or  award  the 
engineering  and  manufacturing  development  contract  for  the  Block  n  system. 
No  DAB  milestone  review  was  held,  and  none  is  planned  until  1997,  although 
the  Program  had  been  in  development  since  1983  and  had  cost  about 
$1.4  billion  through  1992.  The  DAB  program  review  planned  for  March  1993 
did  not  include  the  materiel  release  decision  for  Block  I  and  did  not  ensure 
preparation  of  all  documentation  needed  for  a  thorough  review. 

We  recommended  that  a  DAB  Milestone  n  review  of  the  ASAS  Program  be 
conduc^  and  Aat  remaining  FY  1993  ASAS  funding  be  withheld  until  an 
acquisition  decision  memorandum  was  issued  following  Ae  Milestone  n  review. 
The  USD(A)  partially  concurred  with  our  finding  and  recommendations  and  the 
Comptroller  of  the  Department  of  Defense  concurred. 


Report  No.  93-069,  "The  Airborne  Self-Protection  Jammer  Program  as  a 
Part  of  the  Audit  of  the  Defense  Acquisition  Board  Review  Process— 
FY  1993,"  March  22, 1993.  This  report  found  ftat 

o  The  136  ASPJ  units  being  acquired  under  low-rate  initial  production 
contracts  and  the  20  units  acquired  for  operational  testing  did  not  meet  user 
needs.  The  report  was  classified  as  "Secret"  because  portions  of  the  operational 
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test  and  evaluation  report,  on  which  the  audit  results  were  based,  were  classified 
as  secret.  The  classified  portions  of  the  audit  report  detail  the  reasons  for  which 
the  ASPJ  did  not  meet  user  needs. 

o  A  COEA  had  not  been  performed  to  support  the  planned  Milestone 
in.  Production  and  Development,  decision.  Without  a  COEA,  viable 
alternatives  to  this  new  development  program  were  not  adequately  assessed  as  to 
their  relative  costs  and  operational  effectiveness. 

We  recommended  that  the  program  be  canceled,  the  Navy  and  the  Air  Force 
prepare  one  formal  COEA,  a  Defense  Acquisition  Board  program  review  be 
convened  to  determine  the  best  action  for  fulfilling  overall  DoD  operational 
requirements  for  aircraft  self-protection,  and  the  Navy  conclude  further 
procurement  of  ASPJ  systems  as  low-rate  initial  production  and  terminate 
ongoing  production  activity.  The  Office  of  the  USD(A)  commented  that  the 
recommendations  in  the  draft  report,  issued  December  4,  1992,  were 
accommodated  by  the  Navy's  termination  of  production  contracts,  which 
occurred  on  December  15,  1992.  Potential  monetary  benefit  of  $485.5  million 
from  reduced  procurement  costs  were  realized  as  a  result  of  this  management 
action.  The  Office  of  the  USD(A)  also  agreed  a  COEA  and  DAB  review  would 
be  required  if  the  Navy  or  the  ^r  Force  were  to  restart  any  new  program  to 
meet  their  self-protection  requirements. 


Report  No.  92-104,  "Review  of  the  M1A2  ABRAMS  Tank  Program  as  a 
Part  of  the  Audit  of  the  Effectiveness  of  the  Defense  Acquisition  Board 
Review  Process-FY  1S>92,"  June  16, 1992.  This  report  found  that 

o  The  number  of  ABRAMS  tanks  in  inventory  and  on  contract  exceeded 
requirements.  As  a  result,  plans  to  acquire  an  additional  18  MlAl  tanks  were 
unjustified.  Also,  exercising  an  option  on  the  existing  MlAl  procurement 
contract  to  produce  the  last  62  tanks  in  the  M1A2  configuration  had  not  been 
adequately  justified.  The  Army  could  potentially  save  $24.5  million,  improve 
industrial  mobilization  capabilities,  and  acquire  tanks  suitable  for  entry  into 
service  by  producing  the  units  in  the  MlAl  configuration. 

o  A  DAB  Milestone  IV,  Major  Modification,  review  had  not  been 
schooled  for  the  ABRAMS  tank  program  to  assess  alternatives  for  upgrading 
earlier  configurations  of  Ml  tanks.  The  Army  extensively  documented 
operational  requirements  that  supported  the  incorporation  of  the  MlAl 
configuration,  and  subsequently  the  M1A2  configuration,  into  production  before 
the  M1A2  production  program  was  curtailed  m  April  1990.  However,  the 
Army  had  not  requested  a  DAB  Milestone  IV  review  of  alternatives  for 
upgrading  existing  tanks,  including  affordability  assessments  and  Joint 
Requirements  Oversight  Council  evaluation  of  an  updated  and  validated  system 
threat  assessment  report.  Documentation  required  to  support  program  decisions 
was  also  not  prepar^. 

We  recommended  that  plans  to  modify  the  FY  1991  MlAl  tank  production 
contract  to  substitute  the  production  of  62  M1A2  tanks  for  62  MlAl  tanks  be 
canceled  and  that  a  contract  for  production  of  18  MlAl  tanks  using 
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congressional  supplemental  funding  to  the  FY  1992  Appropriations  Act  not  be 
awarded.  In  addition,  we  recommended  that  a  Milestone  IV  review  of  the 
ABRAMS  tank  program  be  conducted.  The  USD(A)  did  not  respond  to  the 
draft  report  and  did  not  initiate  any  actions  to  mediate  tfie  audit  results  before 
the  contract  modification  for  62  M1A2  tanks  was  executed  on  May  18,  1992. 


Report  No.  92-097,  "F/A-IS  E/F  Program  as  a  Part  of  the  Audit  of  the 
Effectiveness  of  the  Defense  Acquisition  Board  Review  Process— FY  1992," 
June  5,  1992.  This  report  found  that  the  Navy  had  not  performed  a  COEA 
supporting  the  Milestone  IV/II  review  to  enter  the  Engineering  and 

Manufacturing  Development  phase  of  the  acquisition  cycle.  Without  a  COEA, 
viable  alternatives  to  this  new  development  program  may  not  have  been 

adequately  assessed  with  regard  to  their  relative  cost  and  operational 

effectiveness.  The  lack  of  a  COEA  supporting  the  Engineering  and 

Manufacturing  Development  decision  was  conti^  to  die  intent  of  congressional 
direction  on  the  F/A-18  E/F  Program  and  DoD  regulations.  We  recommended 
that  a  formal  COEA  of  the  F/A-18  E/F  and  alternative  programs  be  prepared 
and  program  cost  estimates  and  affordability  assessments  ^  updat^.  The 
USD(A)  did  not  formally  respond  to  the  draft  report.  The  Assistant  Secretary 
of  the  Navy  (Research,  Development  and  Acquisition)  responded  but  did  not 
concur  with  our  finding  and  recommendation.  However,  as  a  result  of  the 
mediation  process,  we  were  provided  documentation  that  sufficient  funding  had 
subs^uently  been  programmed  for  the  F/A-18  E/F  program  and  that  a  COEA, 
meeting  the  intent  of  our  recommendation,  had  been  made  in  response  to  our 
report.  This  COEA  included  the  F/A-18  E/F,  the  current  F/A-18 
configuration,  and  the  AX  Program. 


Report  No.  92-089,  "Cost-Effectiveness  Analyses  for  the  Air  Force  C-17 
Program,"  May  12,  1992.  This  report  found  that  the  cost-effectiveness 
analyses  of  alternatives  to  the  C-17  aircraft  adequately  supported  continuation  of 
the  120-aircraft  C-17  Program.  However,  OSD  and  Air  Force  an^yses  were 
limited  since  performance  of  a  C-141  service-life  extension  program,  as  a 
complement  to  the  120-aircraft  C-17  Program,  was  not  ad^uately  considered  to 
reduce  long-term  airlift  costs  associated  with  fulftlling  mission  needs  identified 
in  the  mobility  requirements  study  directed  by  Congress.  Those  needs  can 
neither  be  met  by  the  120-aircraft  C-17  Program  alone  nor  by  an  expanded  C-17 
Program.  We  recommended  that  a  COEA  be  performed  and  a  special  Defense 
Acquisition  Board  program  review  of  the  C-17  Program  be  conducted  before  the 
Lot  V  production  decision.  We  also  recommended  that  a  service-life  extension 
program  for  the  C-141  aircraft  fleet  be  assessed  as  part  of  the  COEA  to  folfill 
the  requirements  in  the  mobility  requirements  study.  The  USD(A)  and  the 
Assistant  Secretary  of  the  Air  Force  (Acquisition)  nonconcurred  with  our 
finding  and  recommendations  in  the  draft  report.  However,  Congress 
incoriwrated  language  in  the  National  Defense  Authorization  Act  for  FY  1993 
directing  the  Air  Force  to  comply  with  our  recommendations. 


Report  No.  92-074,  "Contracting  Practices  for  the  C-17  Flight  Test 
Aircraft,"  April  10,  1992.  This  report  found  that  the  Air  Force 
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inappropriately  established,  priced,  and  acc^ted  the  "T-1  Assembly  Complete" 
contract  line  item,  which  was  a  prerequisite  for  the  award  of  die  third 
production  lot  of  four  aircraft  that  r^uired  USD(A)  approval.  The  Air  Force 
actions  impaired  effective  OSD  program  oversight  in  die  decision  to  proceed 
with  the  third  production  lot.  As  a  result,  Defense  acquisition  executive 
oversight  of  the  C-17  Program  was  negatively  affected  and,  at  the  time  of 
acc^tance,  the  Government  paid  more  for  "T-1  Assembly  Complete"  than 
justified,  thereby  increasing  program  risk.  Based  on  the  acceptance  of  "T-1 
Assembly  Complete,"  Douglas  liquidated  $1,635  billion  in  previously  made 
progress  payments  and  was  paid  an  additional  $16  million.  No  withholding  was 
made  for  work  not  complete.  We  recommended  that  procedures  be  estabUshed 
to  ensure  that  OSD  staff  are  informed  of  contractual  events  that  affect  Defense 
acquisition  executive  decisions  and  that  OSD  staff  participate,  as  appropriate,  in 
the  accqitance  of  those  events.  The  nature  and  extent  of  this  oversight  should 
be  coordinated  by  a  single  point  of  contact  within  the  Office  of  the  USD(A). 
The  USD(A)  did  not  provide  comments  to  the  draft  report.  The  Assistant 
Secretary  of  the  Air  Force  (Acquisition)  did  not  agree  that  the  degree  of  the 
Office  of  the  USD(A)  participation  should  be  determined  when  the  event-based 
milestones  are  established,  ’^e  Assistant  Secretary  also  nonconcurred  with  our 
statements  that  the  award  of  Lot  HI  required  USD(A)  approval  and  withholds 
from  payment  should  have  been  made  at  the  time  of  "T-1  Assembly  Complete." 
The  Director  of  Procurement  nonconcurred  with  the  recommendation  that  OSD 
staff  participate  in  contract  line  accq)tances  that  may  result  in  Defense 
acquisition  executive  decisions.  The  Director  also  nonconcurred  with  the 
recommendation  that  the  Office  of  the  USD(A)  should  develop  a  process  for 
coordinating  its  participation  at  contractual  events.  We  agreed,  based  on  the 
responses  to  our  recommendations,  that  no  management  action  would  be 
necessary  pending  further  evaluation  of  the  problem  area  in  an  on-going  and 
more  detailed  audit  of  the  low-rate  initial  production  process. 


Report  No.  92-050,  "Review  of  the  Sensor  Rized  Weapon  as  a  Part  of  the 
Audit  of  the  Effectiveness  of  the  Defense  Acquisition  Board  Review  Process- 
FY 1992,"  February  18,  1992.  This  report  found  that  proposed 

program-sj^ific  exit  criteria  were  not  established  in  the  DAB  documentation  to 
proceed  with  future  production  decisions.  As  a  result,  management  did  not 
have  an  effective  means  of  measuring  progress  toward,  or  justifying  proceeding 
with,  future  production  decisions.  In  addition,  acquisition  program  baselines 
did  not  contain  descriptive  performance  parameters  for  lethality  and  launch 
aircraft  survivability.  Also,  the  USD(A)  did  not  approve  the  baseline 
agreement.  As  a  result,  the  developmental  baseline  may  not  provide 
decisionmakers  sufficient  insight  into  achievement  of  essential  system 
performance  characteristics.  We  recommended  the  establishment  of  specific 
exit  criteria  for  future  SFW  production  decisions  and  acquisition  program 
baselines  for  the  number  of  kills  per  aircraft  p^s  required  in  a  countermeasures 
environment  and  the  survivability  of  launch  aircraft  employing  the  SFW.  The 
USD(A)  concurred  and  the  Air  Force  partially  concurred  with  the 
recommendation  to  establish  specific  exit  criteria.  Both  USD(A)  and  the  Air 
Force  partially  concurred  wiA  the  recommendation  to  establish  acquisition 
program  baselines. 
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Report  No.  92-018,  "Air  Force  Short  Range  Attack  Missile  n  Progr^," 
December  12, 1991.  This  report  found  that  the  Air  Force  did  not  have  a  viable 
acquisition  strategy  for  the  continued  development  and  production  of  the 
SRAM  n  Program.  Significant  cost  overruns,  schedule  delays,  performance 
problems,  and  changes  in  requirements  had  invalidated  the  original  strategy. 
The  Air  Force  fail^  to  take  corrective  action  when  the  contractor  failed  to 
make  satisfactory  progress.  In  addition,  the  DAB  had  not  reviewed  the  program 
and  approved  a  revised  acquisition  strategy,  although  piecemeal  approve  of 
acquisition  program  baselines  had  occurred.  Development  costs  were  estimated 
to  exceed  the  contract  ceiling  price  by  as  much  as  $265  million,  the  contractor 
was  performing  to  schedules  that  were  not  contractually  binding,  production 
options  could  have  elapsed  causing  an  estimated  increase  of  $275  million  in 
production  costs,  and  potential  reductions  in  performance  requirements  were 
possible.  We  withdrew  our  recommendation  because  the  President  cancelled  the 
SRAM  II  Program. 
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5000.2  on  Cost  and  Operational 
Effectiveness  Analyses 


A  cost  and  operational  effectiveness  analysis  (COEA)  will  typically  draw  on 
several  sub-analyses.  Those  analyses  include  mission  needs,  threat  and  U.S. 
capabilities,  interrelationship  of  systems,  contribution  of  multi-role  systems, 
measures  of  effectiveness,  costs,  and  cost-effectiveness  comparisons.  The 
following  general  guidelines  apply  to  the  development  of  COEAs. 


1.  Mission  Need  Analysis.  A  mission  need  analysis  assesses  alternatives  in  an 
operational  context,  identifying  what  force  capabilities  would  be  gained  (or 
foregone)  by  pursuing  any  designated  alternatives.  A  mission  ne^  analysis 
assesses  the  strengths  and  weaknesses  of  a  milit^  force  when  confronting  a 
postulated  threat  in  a  speciHed  scenario  or  set  of  circumstances. 

a.  The  scenarios  should  include  a  set  based  on  situations  that  conform  to 
the  scenarios  in  the  Defense  Planning  Guidance,  as  well  as  scenarios  concerning 
U.S.  and  allied  involvement,  and  should  not  conflict  with  the  assumptions  in  the 
Defense  Planning  Guidance  scenarios.  The  analysis  should  address  all  relevant 
situations  in  the  Defense  Planning  Guidance  scenarios.  U.S.  force  availability 
should  be  included  in  the  scenarios  or  established  elsewhere  in  the  Defense 
Planning  Guidance. 

b.  Alternative  cases  may  be  considered  when  they  would  contribute  to 
the  analysis.  In  those  instances,  the  variance(s)  from  the  Defense  Planning 
Guidance  scenario(s)  must  be  clearly  identified  and  addressed. 

c.  Whatever  scenario  is  selected,  the  mission  needs  analysis  must  show 
how  the  alternatives  under  consideration  would  contribute  to  accomplishment  of 
a  national  military  mission  established  by  the  Defense  Planning  Guidance. 

d.  The  COEA  must  describe,  quantitatively  and  qualitatively,  the 
operational  impact  (or  range  of  impacts)  of  responding  to  an  identified 
deficiency  or  opportunity  as  suggested  by  each  alternative  under  consideration. 


2.  Threat  and  U.S.  Capabilities.  The  COEA  must  include  projections  of  the 
enemy  threat.  It  should  describe  the  strengths  and  weaknesses  of  forces  and 
capabilities  that  potential  adversaries  could  employ  in  die  designated  mission 
area  and  show  how  those  forces  and  capabilities  are  projected  to  change. 

a.  Changes  in  the  threat  typically  should  be  examined  at  least  10  years 
into  the  future.  U.S.  capabilities  should  be  typically  projected  at  least  through 
the  end  of  the  6-year  Defense  Program's  funded  delivery  period  and  further  if 
circumstances  warrant. 
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b.  The  evaluation  should  consider  how  U.S.  needs  would  change  as  a 
result  of  changes  in  the  threat.  Additionally,  it  should  address  the  possible 
effects  of  countermeasures  (reactive  or  technologically  feasible)  adversaries 
might  employ  against  the  capabilities  offered  by  each  alternative  being 
evduated. 


3.  System  Interrelationships.  Individual  systems  generally  cannot  be 
evalimted  in  isolation.  Few  deficiencies  can  be  resolved  by  just  one  system,  and 
some  systems  can  complicate  the  use  of  other  friendly  systems.  Therefore,  the 
analysis  must  consider  all  relevant  systems  and  the  synergisms,  such  as 
interoperability,  and  potential  difficulties  they  collectively  represent  on  the 
battlefield. 


4.  Multi-Role  Systems.  A  number  of  systems  can  accomplish  significantly 
different  functions  at  different  times.  For  example,  an  aircraft  carrier  battle 
group  can  support  sea  lane  Defense  operations  against  submarines  one  day  and 
conduct  long-range  power  projection  missions  ashore  the  next.  Accordingly,  as 
appropriate,  a  COEA  should  account  for  this  flexibility  by  investigating  rafter 
than  considering  only  the  outcomes  of  representative  tactical  engagements.  It 
must  also  account  for  occasional  nonavailability  for  one  task  because  of 
application  or  dedication  to  another. 
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Management  Comments 

The  Director,  Acquisition  Program  Integration,  provided  management 
comments  on  the  draft  of  this  report  for  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology.  The  complete  text  of  the  Director's  comments  is 
in  Part  IV  of  this  report.  We  discuss  the  Director’s  comments  on  our 
recommendations  in  Part  n  of  this  rq>ort.  We  discuss  in  this  Appendix  specific 
management  comments  not  addressed  in  Part  n. 

Management  Comments.  The  Director  stated  that  the  draft  of  this  report  did 
not  recognize  their  inputs  to  the  previous  reports  summarized  or  the  results  of 
mediation  meetings. 

Audit  Response.  In  Appendix  C,  we  synopsized  the  audit  reports  on  each 
individual  major  Defense  acquisition  program  reviewed  and  appropriately 
recognized  management  comments  on  each  report  and  mediation  actions. 

Management  Comments.  The  Director  stated  that  Appendix  C  of  the  draft 
report  recognized  that  a  COEA  was  performed  for  the  F/A-18  E/F  aircraft  as  a 
result  of  the  mediation  process  while  the  body  of  the  draft  report  indicated  that  a 
COEA  was  not  performed.  The  Director  idso  stated  that  tiiey  had  previously 
explained  to  us  that  sufficient  analysis  had  been  done  so  that  a  formal  COEA  for 
the  Milestone  IV  decision  was  not  needed. 

Audit  Response.  The  body  of  draft  report  clearly  stated  that  COEAs  were  not 
always  performed  when  required  (underscoring  added  for  emphasis)  or  did  not 
always  adequately  address  viable  alternatives  to  proceeding  with  new  major 
Defense  acquisition  programs.  The  rqwrt  cited  the  F/A-18  E/F  aircraft  as  an 
example  of  such  a  situation.  The  fact  that  a  COEA  was  subsequently  performed 
as  a  result  of  our  audit  rq)ort  does  not  alter  the  fact  that  in  the  case  of  the 
F/A-18  E/F  aircraft,  a  COEA  was  not  performed  before  the  Milestone  IV/n 
DAB  review  as  required  by  DoD  Instruction  5000.2. 

Management  Conunents.  The  Director  stated  that  the  draft  report  ignored  the 
fact  that  a  COEA  was  optional  for  a  Milestone  IV  review  of  the  DDG-51 
destroyer  variant,  per  DoD  Instruction  5000.2.  The  Director  also  felt  that  we 
had  ignored  their  response  to  the  final  report  on  the  DDG-51  that  non-concurred 
with  our  finding  and  the  adequacy  of  the  Navy  Destroyer  Variant  Study  as  a 
COEA. 

Audit  Response.  The  Director  is  correct  that  a  formal  COEA  was  not  required 
for  the  DDG-51  destroyer  variant.  Our  report  recognized  that  fact,  but 
indicated  that  the  alternative  to  the  COEA  that  was  performed  did  not  consider 
all  alternatives  to  the  continued  procurement  of  DDG-51  destroyers.  We 
continue  to  believe  the  Navjr  should  have  conducted  a  COEA  when  the  DDG-51 
program  became  an  acquisition  category  ID  program  in  August  1992.  Also,  we 
disagreed  with  the  decision  of  the  Director,  Program  Analysis  and  Evaluation, 
in  October  1992  to  accept  the  Destroyer  Variant  Study  by  the  Navy  as  a 
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substitute  for  a  formal  COEA  of  the  DDG-51  program.  While  the  Destroyer 
Variant  Study  did  present  several  options  for  a  new  and  improved  DDG-51 
ship,  it  did  not  consider  all  viable  alternatives. 

While  the  USD(A&T)  did  not  concur  with  our  recommendation  for  a  formal 
COEA  for  the  DDG-Sl  destroyer  variant,  we  believe  that  his  subsequent  actions 
were  responsive  to  the  intent  of  the  report  recommendation.  He  halted  DDG-51 
production  with  the  third  ship  scheduled  in  FY  1994  pending  review  of 
documentation  for  the  Milestone  IV  DAB.  He  also  requested  initiation  of  a 
Destroyer  Acquisition  Study  that  required  a  detailed  assessment  of  alternative 
DDG-51  force  levels,  procurement  actions,  and  attendant  cost  and  industrial 
base  implications  to  include  foreign  military  sales  and  use  of  upgrades  versus 
ship  retirements.  Further,  he  brought  up  an  alternative  Surface  Combatant  21 
ship  concept  for  a  Milestone  O  review  to  include  conducting  a  surface 
combatant  COEA  beginning  with  Milestone  I. 

Management  Comments.  The  Director  stated  that  our  report  on  "Contracting 
Practices  for  the  C-17  Flight  Test  Aircraft"  did  not  address  COEA  or  exit 
criteria  was  closed  out  by  a  mediation  meeting  on  September  18,  1992,  and  by 
our  own  wording  recognized  that  "no  management  action  would  be  necessary." 

Audit  Response.  We  summarized  the  rqwrt  on  "Contracting  Practices  for  the 
C-17  Flight  Test  Aircraft"  in  Appendix  C  as  an  example  where  program 
management  had  inade(}uacies  in  ^e  approval  and  monitoring  of  minimum 
accomplishments  and  exit  criteria.  As  far  as  management  actions  needed,  the 
Director's  comments  took  our  report  wording  somewhat  out  of  context.  To 
more  accurately  quote  our  report,  it  stated  that,  "no  management  actions  would 
be  necessa^  pending  further  evaluation  of  the  problem  area  in  an  on-going  and 
more  detailed  audit  of  the  low-rate  initial  production  process."  We  did  not 
concede  that  no  problems  existed,  but  that  management  declined  to  take  action 
on  the  problems  without  additional  audit  effort  on  our  part. 

Management  Comments.  The  Director  stated  that  SRAM  n  should  not  be 
cited  in  the  report  because  the  SRAM  n  predates  current  COEA  policy. 

Audit  Response.  DoD  Instruction  5000.2,  February  23,  1991,  states  that  all 
Defense  acquisition  programs  scheduled  for  milestone  reviews  6  months  after 
issuance  of  the  instruction  are  subject  to  the  new  procedures  and  documentation 
requirements.  Our  report  on  the  SRAM  n  noted  the  need  to  update  the  COEA 
for  the  low-rate  initial  production  decision  scheduled  for  May  1993. 
Accordingly,  the  SRAM  n  program  was  subject  to  DoD  Instruction  5000.2 
when  our  final  report  was  issued  on  December  12,  1991. 

Management  Comments.  The  Director  stated  that  the  ASPJ  program  also 
predated  current  COEA  policy  in  DoD  Instruction  5000.2.  Further,  the 
Director  indicated  that  the  report  wrongly  suggested  that  the  Inspector  General 
saved  DoD  $485  million  as  a  result  of  the  ASPJ  audit,  noting  that  the  claim  was 
dropped  in  mediation  meetings. 

Audit  Response.  As  previously  discussed,  the  current  COEA  policy  is 
applicable  for  all  Defense  programs  scheduled  for  milestone  reviews  after 
August  23,  1991.  Our  report  on  the  ASPJ  program  noted  that  a  COEA  had  not 
been  performed  to  support  the  Milestone  niB,  Production  and  Deployment, 
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decision  scheduled  for  September  14,  1992.  Accordingly,  the  ASPJ  program 
was  subject  to  the  COEA  requirements  in  DoD  Instruction  5000.2  when  our 
ASPJ  report  was  issued  on  March  22,  1993. 

We  did  not  intend  to  claim  monetary  benefit  resulting  from  the  Navy's  action 
terminating  ASPJ  production  contracts.  As  noted  in  our  draft  ASPJ  report, 
issued  December  4,  1992,  the  Navy  planned  to  acquire  240  ASPJ  units  at  a  cost 
of  $485.5  million  from  FYs  1993  through  1998  and  an  additional  304  ASPJ 
units  after  FY  1998  at  an  estimated  cost  of  $623.5  million.  We  recommended 
that  the  Navy  terminate  ASPJ  production  contracts  because  the  ASPJ  had  not 
passed  its  o^rational  testing.  The  Navy  terminated  production  contracts  on 
December  15,  1992.  In  this  rqwrt,  we  stated  Aat  the  Navy  reduced 
procurement  costs  by  $485.5  million  as  a  result  of  its  management  action. 

Management  Comments.  The  Director  stated  that  the  draft  rq>ort  assertion 
that,  "[COEA]  analyses  often  were  taylored  to  support  the  recommended 
alternative"  was  not  based  on  any  substantive  or  convincing  evidence  and  that 
this  assertion  should  be  deleted. 

Audit  Response.  The  report  statement  that  the  COEA  analyses  often  were 
tailored  to  support  the  recommended  alternative  was  based  on  the  results  of 
audit  reported  for  the  C-17,  F/A-18  E/F,  M1A2,  ASPJ,  and  DDG-51  that  were 
summarized  in  Appendix  C. 

Management  Comments.  The  Director  stated  that  COEA  analyses  are  not 
initiate  at  the  DAB  6-month  planning  meeting  but  when  the  program  enters  the 
specific  acquisition  phase. 

Audit  Response.  As  stated  in  the  draft  report,  the  Director,  Program  Analysis 
and  Evaluation,  provides  guidance  for  preparation  of  COEAs  at  planning 
meetings,  which  occur  6  months  before  scheduled  DAB  reviews.  Although 
COEAs  are  sometimes  initiated  before  the  Director,  Program  Analysis  and 
Evaluation,  provides  guidance  on  their  preparation,  this  was  not  the  case  for  the 
C-17  aircraft,  F/A-18  E/F  aircraft,  M1A2  tank,  ASPJ,  or  DDG-51  destroyer 
programs  that  we  reviewed. 

Management  Comments.  The  Director  stated  that  the  draft  report  finding  on 
exit  criteria  contained  very  few  credible,  specific  examples  of  supposed 
weakness  in  the  current  policy.  Further,  he  stated  that  the  M1A2  is  an 
acquisition  category  1C  program  that  did  not  belong  in  an  audit  report  on  the 
effectiveness  of  the  DAB  review  process  and  that  the  SRAM  n  report  predated 
current  exit  criteria  policy. 

Audit  Response.  We  deleted  the  draft  report  finding  to  improve  the  use  of  exit 
criteria  in  view  of  management  comments  that  recommended  changes  would  not 
be  made  without  more  conclusive  examples  of  weaknesses  in  the  current  use  of 
exit  criteria.  Because  we  continue  to  believe  that  the  use  of  exit  criteria  can  be 
significantly  improved,  we  plan  to  initiate  a  separate  audit  of  this  subject  if  it  is 
sml  warranted,  when  current  actions  to  streaMine  the  acquisition  process  are 
completed. 

Our  inclusion  of  the  audit  report  on  the  M1A2  was  appropriate  because  when 
we  selected  the  M1A2  program  for  audit,  it  was  scheduled  for  a  DAB  review  in 
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January  1992.  Although  the  USD(A&T)  subsequently  waived  the  DAB  review, 
our  audit  of  the  adequacy  of  the  program  office's  preparation  for  the  DAB 
review  is  appropriate  for  inclusion  in  this  summary  report. 

Basic  policy  on  the  use  of  exit  criteria  existed  in  DoD  Instruction  5000.2, 
Februa^  23,  1991,  at  the  time  of  our  audit  of  the  SRAM  n  program.  Our 
audit  of  the  SRAM  n  was  made  from  April  through  July  1991  and  the  report 
was  issued  December  12,  1991.  On  February  26,  1993,  DoD  added 
Attachment!,  Section  A,  Part  11  of  DoD  Instruction  5000.2  to  provide 
additional  guidance  on  acquisition  program  baselines  and  exit  criteria. 

Management  Comments.  The  Director  stated  that  after  a  thorough  review  of 
the  required  documentation,  the  Command,  Control,  Communications,  and 
Intelligence  Systems  Committee  conducted  a  Milestone  n  review  of  the  ASAS 
program  August  23,  1993.  Based  on  the  recommendation  of  this  Committee, 
the  USD(A&T)  signed  an  Acquisition  Decision  Memorandum  on  October  21, 
1993,  which  completed  the  "paper  DAB"  for  the  ASAS  program.  The 
Acquisition  Decision  Memorandum  approved  the  Army's  request  to  proceed 
with  the  Engineering  and  Manufacturing  Development  phase  of  the  ASAS  Block 
II  program  and  authorized  the  Comptroller  of  the  Department  of  Defense  to 
release  the  FY  1993  funds  that  had  been  on  withhold  pending  favorable 
completion  of  the  DAB.  Contrary  to  the  assertion  in  our  audit  report,  the 
Director  stated  that  they  reviewed  all  documents  needed  for  the  ASAS  Mlestone 
HDAB. 

Audit  Response.  We  issued  our  audit  rqx>rt  on  the  ASAS  program  April  20, 
1993.  The  actions  that  the  Director  cites  were  taken  after  the  issuance  of  our 
audit  report  and  as  a  result  of  our  recommendations.  Die  synopsis  of  the  report 
that  we  issued,  as  presented  in  Appendix  C,  is  accurate. 

Management  Comments.  The  Director  stated  that  the  summary  on  MIDS  in 
Appendix  C  did  not  recognize  the  difference  between  an  international 
acquisition  program  and  a  DoD  acquisition  program.  The  primary  difference  is 
that  an  international  program  is  driven  by  an  international  Memorandum  of 
Understanding,  which  defines  the  specifics  of  the  cooperative  effort.  Once  the 
program  Memorandum  of  Understanding  is  agreed  upon,  which  often  takes  a 
considerable  amount  of  time,  the  program  must  be  inserted  into  the  U.S. 
acquisition  process.  At  the  time  of  tiie  Inspector  General  review,  tiie  Navy,  as 
the  lead  Service,  was  beginning  to  prepare  MIDS  documentation  for  the 
Milestone  n  DAB.  After  £e  issuance  of  die  audit  report,  the  program  office: 
completed  all  Milestone  n  DAB  requirements;  was  direct^  by  the  USD(A&T) 
to  proceed  with  the  Engineering  and  Manufacturing  Development  phase; 
awarded  the  Engineering  and  Manufacturing  Development  contract  to  the 
international  consortium;  completed  a  study  in  June  1994  to  reduce  program 
cost,  schedule,  and  insert  new  technology;  and  will  request  USD(A&'^  to 
approve  exit  criteria  for  Engineering  and  Manufacturing  Development  in 
accordance  with  the  June  study  results.  Finally,  the  Director  stated  that 
documents  such  as  the  Developmental  Test  and  Evaluation  Rqport  and  Early 
Operational  Assessment  Report  are  required  for  LRIP,  not  Milestone  n. 

Audit  Response.  Our  audit  report  on  the  MIDS  program,  issued  June  21, 
1993,  recognized  r^uirements  for  both  major  Defense  acquisition  programs  and 
intemationd  acquisition  programs  in  detail.  It  was  neither  possible  nor  relevant 
to  cite  such  information  for  the  synopsis  of  the  report  in  Appendix  C.  The 
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actions  that  the  Director  cited  were  taken  after  the  issuance  of  our  audit  report 
and  as  a  result  of  our  recommendations.  The  synopsis  of  the  report  that  we 
issued,  as  presented  in  Appendix  C,  is  accurate. 

Regarding  the  Director's  comment  on  the  Developmental  Test  and  Evaluation 
Report  and  Early  (^rational  Assessment  Report,  we  agreed  in  our  report  that 
the  USD(A&p  should  require  the  Navy  to  prepare  these  reports  before  entry 
into  low-rate  initial  production. 

Management  Comments.  The  Director  stated  that  the  summary  for  the  Sensor 
Fused  Weapon  audit  presented  in  Appendix  C  wrongly  suggested  that  exit 
criteria  for  low-rate  initial  production  were  not  well-planned. 

Audit  Response.  The  Director  provided  no  information  to  refute  the  facts  on 
the  Sensor  Fused  Weapon  as  summarized  in  our  audit  report.  ProjMsed 
program-specific  exit  criteria  for  the  Sensor  Fused  Weapon  were  not  estabbshed 
in  the  DAB  documentation  to  proceed  with  fiiture  Sensor  Fused  Weapon 
production  decisions. 

Management  Comments.  The  Director  stated  that  the  policy  of  DoD 
Instruction  5000.2  is  that  exit  criteria  be  flexible  enough  to  aUow  the  project 
manager  to  execute  key  events  without  the  formality  of  a  review  by  the 
milestone  decision  authority.  The  director  felt  that  our  draft  finding  ignored 
that  policy. 

Audit  Response.  We  specifically  recognized  the  cited  policy  in  the  draft  report 
finding  on  exit  criteria. 
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Appendix  F 

Recommendation 

Reference 

1. 

2. 

3. 

4. 


Summary  of  Potential 
Resulting  From  Audit 


Description  of  Benefit 


Internal  Control.  Will  provide 
sufficient  time  for  preparation  of  the 
COEAs,  which  are  fundamental  to 
sound  acquisition  decisions. 

Internal  Control.  Update  of 
COEAs,  when  needed  as  a  result  of 
revisions  to  acquisition  program 
baselines,  can  limit  the  need  for 
DAB  program  reviews  or  other 
forms  of  oversight. 

Program  Results.  Will  ensure  that 
the  COEA  will  be  done  properly. 

Program  Results.  Will  ensure  that 
the  COEA  has  been  done  properly. 


Benefits 

Amount  and/or 
Type  of  Benefit 

Nonmonetary. 

Nonmonetary. 

Nonmonetary. 

Nonmonetary. 


Appendix  G,  Organizations  Visited  or  Contacted 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology,  Washington,  DC 
Under  Secretary  of  Defense  (Comptroller),  Washington,  DC 
Assistant  Secretary  of  Defense  for  Command,  Control,  Communications,  and 
Intelligence,  Washington,  DC 
Director,  Defense  Procurement,  Washington,  DC 
Director,  Defense  Research  and  Engineering,  Washington,  DC 
Director,  Operational  Test  and  Evaluation,  Washington,  DC 
Director,  Acquisition  Program  Integration,  Washington,  DC 
Director,  Program  Analysis  and  Evaluation,  Washington,  DC 


Department  of  the  Army 

Chief  of  Staff,  Army,  Washington,  DC 

Deputy  Chief  of  Staff  for  logistics,  Washington,  DC 
Deputy  Chief  of  Staff  for  Operations  and  Plans,  Washington,  DC 
Assistant  Secretary  of  the  Army  for  Financial  Management,  Washington,  DC 
Assistant  Secretary  of  the  Army  for  Research,  Development  and  Acquisition, 
Washington,  DC 

Director,  Army  Cost  and  Economic  Analysis  Center,  Arlington,  VA 
Commander,  U.S.  Army  Missile  Command,  Redstone  Arsenal,  AL 
Commander,  U.S.  Army  Tank-Automotive  Command,  Warren,  MI 
Army  Materiel  Systems  Analysis  Activity,  Army  Materiel  Command,  Aberdeen,  MD 


Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  for  Financial  Management,  Washington,  DC 
Assistant  Secretary  of  the  Navy  for  Research,  Development  and  Acquisition, 
Washington,  DC 

Marine  Corps  Systems  Command,  Quanitco,  VA 

Naval  Air  Systems  Command,  Arlington,  VA 

Naval  Sea  Systems  Command,  Arlington,  VA 

Space  and  Naval  Warfare  Systems  Command,  Arlington,  VA 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  for  Acquisition,  Washington,  DC 

Director,  Long  Range  Power  Projection,  Special  Operations  Forces,  and  Airlift 
Programs,  Washington,  DC 
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Department  of  the  Air  Force  (Continued) 

Assistant  Secretary  of  the  Air  Force  for  Financial  Management  and  Comptroller, 
Washington,  DC 

General  Counsel  of  the  Air  Force,  Washington,  DC 
Air  Mobility  Command,  Scott  Air  Force  Base,  IL 
Air  Force  Materiel  Command,  Wright-Patterson  Air  Force  Base,  OH 
Aeronautical  Systems  Center,  Wright-Patterson  Air  Force  Ba^,  OH 
C-17  System  Program  Office,  Wright-Patterson  Air  Force  Base,  OH 
Sensor  Fuzed  Weapon  and  Airfield  Attack  System  Program  Office,  Eglin  Air 
Force  Base,  FL 

Short  Range  Attack  Missile  n  System  Program  Office,  Wright-Patterson  Air 
Force  Base,  OH 

Deputy  Assistant  Secretary  of  the  Air  Force  for  Contracting,  Washington,  DC 
Headquarters,  Wamer-Robins  Air  Logistics  Center,  Robins  Air  Force  Base,  GA 
Air  Force  Cost  Analysis  Agency,  Washington,  DC 
Air  Force  Safety  Agency,  Norton  Air  Force  Base,  CA 

Commander,  Air  Force  Operational  Test  and  Evaluation  Center,  Eglin  Air  Force 
Base,  FL 


Defense  Agencies 

Defense  Intelligence  Agency,  Washington,  DC 

Defense  Mapping  Agency  Aerospace  Center,  St.  Louis,  MO 

Defense  Contract  Audit  Agency,  Boeing  Aerospace  and  Electronics,  Seattle,  WA 

Defense  Contract  Audit  Agency,  Douglas  Aircraft  Company,  Long  Beach,  CA 


Other  Defense  Organizations 

The  Joint  Staff,  Washington,  DC 

Defense  Information  Systems  Agency,  Arlington,  VA 

Defense  Plant  Representative  Office,  Boeing  Aerospace  and  Electronics,  Seattle,  WA 
Defense  Plant  Representative  Office,  Douglas  Aircraft  Company,  Long  Beach,  CA 
Defense  Plant  Representative  Office,  LocWieed-Georgia  Company,  Marietta,  GA 


Non-DoD  Federal  Organization 

Senate  Committee  on  Armed  Services,  Washington,  DC 


Non-Governmental  Organizations 

Boeing  Aerospace  and  Electronics,  Seattle,  WA 
McDonnell  Douglas  Corporation,  Long  Beach,  CA 
McDonnell  Douglas  Corporation,  St.  Louis,  MO 
Lockheed-Georgia  Com^y,  Marietta,  GA 
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Appendix  H.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 
Under  Secretary  of  Defense  (Comptroller)  _ 

Deputy  Under  Secretary  of  Defense  (Acquisition  Reform) 

Assistant  Secretary  of  Defense  for  Command,  Control,  Communications,  and 
Intelligence 

Director,  Defense  Procurement 
Director,  Defense  Research  and  Engin^ring 
Director,  Operational  Test  and  Evaluation 
Director,  Acquisition  Program  Integration 
Director,  Program  Analysis  and  Evaluation 


Department  of  the  Army 

Secretaiy  of  the  Army 
Chief  of  Staff,  Army 

Deputy  Chief  of  Staff  for  Logistics 
Deputy  Chief  of  Staff  for  Operations  and  Plans 
Assistant  Secretary  of  the  Army  for  Financial  Management 
Assistant  Secretary  of  the  Army  for  Research,  Development  and  Acquisition 
Auditor  General,  Department  of  the  Army 
Director,  Army  Cost  and  Economic  Analysis  Center 
Commander,  U.S.  Army  Missile  Command 
Commander,  U.S.  Army  Tank-Automotive  Command 
Army  Materiel  Systems  Analysis  Activity,  Army  Materiel  Command 


Department  of  the  Navy 

Secretary  of  the  Navy 

Assistant  Secretary  of  the  Navy  for  Financial  Management 

Assistant  Secretary  of  the  Navy  for  Research,  Development  and  Acquisition 

Comptroller  of  the  Navy 

Auditor  General,  Department  of  the  Navy 

Marine  Corps  Systems  Command 

Naval  Air  Systems  Command 

Naval  Sea  Systems  Command 

Space  and  Naval  Warfare  Systems  Command 
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Department  of  the  Air  Force 

Secretary  of  the  Air  Force 

Assistant  Secretary  of  Air  Force  for  Acquisition 

Assistant  Secretary  of  Air  Force  for  Financial  Management  and  Comptroller 
General  Counsel  of  the  Air  Force 
Auditor  General,  Department  of  the  Air  Force 
Air  Force  Materiel  Command,  Wright-Patterson  Air  Force  Base,  OH 
Aeronautical  Systems  Center,  Wright-Patterson  Air  Force  Base,  OH 
Commander,  Air  Force  Operational  Test  and  Evaluation  Center,  Eglin  Air  Force 


Other  Defense  Organizations 

Director,  Defense  Contract  Audit  Agency 

Director,  Defense  Logistics  Agency 

Director,  National  Security  Agency 

Inspector  General,  Central  Imagery  Office 

Inspector  General,  Defense  Intelligence  Agency 

Inspector  General,  National  Security  Agency 

Director,  Defense  Logistics  Studies  Information  Exchange 


Non-Defense  Organizations  and  Individuals 

Office  of  Management  and  Budget 

U.S.  Gener^  Accounting  Office,  National  Security  and  International  Affairs  Division 
Technical  Information  Center 

Chairn^  and  Raking  Minority  Member  of  the  following  Congressional  Committees 
and  Subcommittees: 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Operations 

House  Subcommittee  on  Legislation  and  National  Security,  Committee  on 
Government  Operations 
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Part  IV  -  Management  Comments 


Director,  Acquisition  Program 
Integration,  Comments 


Director,  Acquisition  Program  Integration,  Comments 


PRINCIPAL  DEPUTY  UNDER  SECRETARY  OF  DEFENSE 

aOISOCFCNW  KNTAOON 
WASHINGTON,  DC  SM0I-30IB 


7  JUNB94 


AFX/ASM 


MEMORAWDUM  FOR  DIRECTOR,  ACQUISITION  MANAGEMENT  DIRECTORATE 

SUBJECT:  Dsfense  Acquisition  Board  Raviav  Process  (Project 
No.  2AE-0033) 


We  have  revieved  your  draft  susaary  report  dated 
March  29,1994,  and  offer  the  follovinq  overall  response  to  your 
findings  and  reconnendations. 

We  do  not  believe  that  your  findings  and  reconaendations 
contribute  very  such  to  the  effectiveness  of  the  DAB  process. 

We  are  particularly  disturbed  when  you  sinply  staple  inaccurate 
summaries  of  previous  audits  together  with  little  apparent 
analysis.  Your  alleged  ••data"  not  only  ignore  our  inputs  to 
those  previous  reports,  but  also  sees  to  ignore  the  results  of 
the  mediation  meetings  that  occurred. 

We  feel  in  the  case  of  COEAs  that  you  have  added  some 
value  since  we  generally  concur  in  your  recommendations  (i.e. 
we  only  non-concur  in  one  of  the  four  recommendations  and 
partially  concur  in  two  others).  However,  in  the  case  of  exit 
criteria  you  are  simply  not  making  a  good  case  to  change  5000 
policy  as  we  non-concur  in  five  of  the  six  recommendations. 

You  are  proposing  to  add  more  levels  of  complexity  to  the 
bureaucracy  that  this  Administration  is  attempting  to  reduce. 

Our  detailed  comments  on  the  findings  and  recommendations 
are  attached. 


Gene  H.  Porter 
Director,  Acquisition 
Program  Integration 


Attachment 


AcamamoN  AND 
nCMMOLiMY 


O 
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Attachaent 


L&2JU — COft  tad  OPtrattQBal  Iffactivaptii  Analvaig 

Ha  quastion  your  auanary  findings  ragarding  r/A-18  E/P, 

DDG-51,  C-17,  SRAM  II,  and  ASPJ: 

-  Your  Appendix  c  suBsary  of  tha  F/A-18  E/F  audit  notes 
that  as  a  result  of  tha  aadiation  process  a  COEA  had 
bean  parfomad.  This  contradicts  your  finding  on 
page  9  that  a  COEA  was  Qpt  parfoned.  As  va  have 
previously  explained ,  sufficient  analysis  had  already 
been  done  so  that  a  foraal  COEA  for  tha  Milestone  IV 
decision  was  not  needed. 

*  Your  Appendix  C  suanary  of  the  DDG-51  and  the 
findings  on  page  9  ignore  the  fact  that  a  COEA  is 
optional  for  a  Milestone  IV  per  5000.2  policy.  You 
have  chosen  to  ignore  our  Feb  24,  1994,  detailed 
response  which  non-concurred  with  your  assertion 
regarding  the  DDG^-Sl  COEA  and  adequacy  of  the  Navy 
Destroyer  Variant  study  as  a  COEA. 

*  Your  Appendix  C  includes  a  suanary  of  Report  No.  92* 
074  on  C*17  contracting  practices.  This  report  does 
llfit  address  COEA  or  exit  criteria;  was  closed  out  by 
the  Sep  18,  1992,  mediation  meeting;  and  by  your  own 
wording  notes  that  "no  management  action  would  be 
necessary."  This  section  is  simply  an  inert  filler 
in  your  report. 

*  The  Appendix  C  suaaary  and  finding  on  page  9 
regarding  the  SRAM  II  do  not  belong  in  this  report  as 
the  SRAM  II  predates  current  COEA  policy. 

*  In  like  manner  the  ASFJ  is  a  mature  program  (now 
terminated)  that  predates  current  COEA  5000.2  policy. 
Appendix  C  also  wrongly  suggests  that  the  IG  has 
saved  $4a5M  as  a  result  of  their  ASPJ  audit.  This 
claim  was  dropped  by  the  IG  in  mediation  meetings  but 
reappears  in  this  ASPJ  summary.  This  summary  also 
doesn't  belong  in  this  report. 


Your  finding  on  page  13  that  "[COEA]  analyses  often 
were  tailored  to  support  the  recommended  alternative" 

Is  not  based  on  any  substantive  or  convincing  evidence. 
This  assertion  should  be  deleted. 


Director,  Acquisition  Program  Integration,  Cmnments 


•  COEA  mnalyses  ara  not  initiatad  at  tha  DAB  six-nonth 
planning  meeting  but  %rhen  tha  prograa  antara  tha 
apecific  acquisition  phase. 


l!hia  finding  is  particularly  fliaay  vith  vary  fav 
credible,  specific  axaaples  of  supposed  vaaknass  In  the  currant 
policy  docunented  in  the  finding  or  in  Appendix  C.  We  also  see 
vary  little  connection  between  the  findings  and  tha  resultant 
racoamendations.  We  take  issue  vith  your  findings  regarding 
inadequacy  of  exit  criteria  for  SFW,  MIA2,  C-17,  SRAM  II,  and 
NIDS.  We  did  respond  in  writing  to  your  audit  and  mediation  on 
MIA2.  The  KIA2  is  an  ACAT  1C  so  it  doesn't  belong  in  a  report 
on  the  effectiveness  of  the  DAB.  The  ARMY  does  have  exit 
criteria  for  tha  MIA2.  Your  Appendix  C  does  not  address  any 
problem  vith  c*17  exit  criteria  so  it  is  not  clear  why  it  is  in 
the  main  finding.  The  SRAM  IX  predates  currant  exit 
criteria  policy. 

We  find  significant  errors  in  Appendix  C  regarding  A5AS 
and  HIDS.  After  a  thorough  raviav  of  tha  required  DoD  5000 
documentation,  a  C3X  Systems  Committee  Milestone  II  Review  of 
the  ASAS  program  was  conducted  on  August  23,  1993.  Based  on 
the  recommendation  of  the  C3I  Systems  Committee,  the  Under 
Secretary  of  Defense  (Acquisition  and  Technology)  signed  the 
ASAS  Program  Acquisition  Decision  Memorandum  (ADM)  on 
October  21,  1993,  which  completed  the  "paper  DAB"  for  the  ASAS 
program.  The  ADM  approved  the  Army's  request  to  proceed  vith 
the  Engineering  and  Manufacturing  Development  (EMD)  phase  of 
the  ASAS  Block  II  program  and  authorized  the  DoD  Comptroller  to 
release  the  FY  93  funds  that  had  been  on  withhold  pending 
favorable  completion  of  the  DAB.  Contrary  to  your  assertion  we 
reviewed  all  documents  needed  for  the  ASAS  Milestone  II  DAB. 

The  summary  on  HiDs  in  Appendix  C,  does  not  recognize  the 
difference  between  an  international  acquisition  program  and  a 
DoD  acquisition  program.  The  primary  difference  is  that  an 
international  program  is  driven  by  an  international  Memorandum 
of  Understanding  (MOU) ,  which  defines  the  specifics  of  the 
cooperative  effort.  Once  the  program  MOU  is  agreed  upon,  which 
often  takes  a  considerable  amount  of  time,  the  program  must  be 
inserted  into  the  US  acquisition  process.  MIDS-LVT,  an  ACAT  1 
program,  was  determined  by  the  C3I  Systems  Committee  to  be 
ready  for  a  Milestone  II  DAB  decision.  At  the  time  of  the  XG 
review,  the  Navy,  as  the  lead  Service,  was  beginning  the 
process  of  preparing  MIDS*»LVT  documentation  for  the  Milestone 
II  DAB.  Since  the  MID5-LVT  IG  report  was  concluded,  the 
program:  has  completed  all  Milestone  IX  DAB  requirements;  has 
been  directed  by  USD(A(T)  to  proceed  vith  the  EMD  phase;  has 
awarded  the  EMD  contract  to  the  international  consortium 
MIDSCO;  will  complete  a  study  in  June  1994  to  reduce  program 
cost,  schedule,  and  insert  new  technology;  and  will  request 
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costr  schedule,  and  insert  new  technology;  and  will  request 
USD(A&T)  to  approve  exit  criteria  for  EMO  in  accordance  with 
the  June  study  results.  We  pointed  out  earlier  that  the 
"missing  documents"  (i.e.,  DT&£  report  and  early  operational 
assessment  are  required  for  LRIP,  ApI  Milestone  II. 

The  Appendix  C  summarizing  the  Sensor  Fuzed  Weapon  audit 
tnrongly  suggests  that  exit  criteria  for  LRIP  were  not  planned. 
This  mistake  was  probably  due  to  your  inappropriate  review  of 
draft  documentation  before  the  DAB  process  had  been  concluded. 

Our  5000.2  policy  states  that  exit  criteria  are  to  l>e 
flexible  enough  to  allow  the  FM  to  execute  key  events  without 
the  formality  of  MDA  review.  Your  finding  ignores  that. 

In  summary,  your  basic  finding  is  not  well  substantiated 
and  your  Appendix  C  paints  an  incorrect  picture  that  ignores 
our  response  to  draft  IG  reports  and  the  resultant  mediation. 
This  situation  casts  doubt  on  your  basic  rationale  for  changing 
current  exit  criteria  policy. 


EtfigMsadstiTB  111 

We  concur  in  your  reccomendation  to  more  cleanly  start 
COEAs  12  months  before  the  DAB  (as  a  normal  procedure). 


Becommendation  A. 2 

We  strongly  non-concur  with  your  recommendation  to 
automatically  update  a  COEA  when  APB  breaches  occur.  Given  the 
sheer  number  of  APB  breaches,  implementing  this  recommendation 
would  shut  down  our  entire  acquisition  process.  The  nature  and 
magnitude  of  the  breach  may  not  necessitate  a  COEA  update  which 
could  take  6  to  12  months  to  complete.  This  is  massive 
overkill  that  violates  the  spirit  of  the  National  Performance 
Review  which  found  an  overwhelming  number  of  complex 
regulations. 


Rscommendation  A. 3 

We  partially  concur  with  your  recosuaendation  regarding 
early  involvement  of  the  JROC  in  the  COEA  process.  While  we 
feel  that  it  irould  be  useful,  the  JROC  did  not  respond  to  us  on 
this  recommendation.  We  need  to  discuss  this  with  the  JROC. 


Recommendation  A.4 

We  partially  concur  with  your  recommendation  to  have  the 
JROC  provide  a  COEA  assessment  to  PASE.  PA&E  tells  us  that 
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they  would  wwleoM  tlw  JROC  MsoasMot  but  w«  again  naad  to 
diacusa  thia  with  tha  JXOC. 


Kacpa^ndation  B.l.a 

non-concut  In  tha  racoMMndatlon  to  hava  BAE  approval 
of  ravialona  to  axlt  eritaria  on  DAB  (ACAT  10)  prograu.  Ha  do 
not  ballava  that  it  ia  appropriata  for  a  coaiponant  aeguiaition 
axecutive  to  approve  reviaiona  to  DAB  exit  criteria. 

Satia  faction  of  axit  criteria  foma  the  baaia  for  entering  the 
next  phaae  of  development  and  frequently  trlggera  a  aignifieant 
obligation  of  funda.  Hhile  it  may  be  neceaaary  to  modify  exit 
criteria  auch  modifieatlona  abould  be  approved  only  by  the 
adleatone  decialon  authority,  OSD(MT). 


Kecommandation  B.l.b 

He  partially  concur  in  thia  recommendation  regarding  the 
PEO  approving  corrective  action  plana  to  addreaa  exit  criteria 
problema  areaa,  but  aee  no  reaaon  for  thia  to  be  in  a  top  level 
DoD  policy  document.  The  dutiaa  of  a  PEO  to  approve  action 
plana  ia  inherent  in  bin  aenior  level  poaition. 


■■'’"-"TlflliioB  B.l.e 

He  non-concur  in  PEO  approval  of  ACAT  20  exit  criteria 
attainment  for  the  aame  reaaona  aa  B.l.a  above. 


■KacOmmeBdatloni  n.l.d.  n.T.a.  and  n.T.h 

He  non-concur  in  theae  recoaoundationa  that  apecifically 
require  DAES  reporting  on  exit  criteria.  He  carefully 
conatructed  the  5000.2  policy  to  allow  exit  criteria  to  be 
crafted  ao  that  deciaiona  could  be  made  that  did  not 
automatically  require  a  OSD(AtT)  raviaw.  He  did  not  want  to 
create  more  OSD  overaight  but  wanted  the  PM  to  aet  hia  own 
internal  criteria  to  be  met  within  a  phaae.  Tour 
recommendationa  illuatrata  again  your  ovarly  eimpliatie 
approach  to  aeguiaition  ovoralght.  Every  anred  of  program 
information  la  not  reported  in  the  DABS,  ouat  beciuae 
aomething  ia  not  in  the  DAES  doea  not  mean  that  OSD  doean't 
know  ita  atatua.  Tou  ahould  nnderatand  that  many  exit  criteria 
are  not  eaaily  reaponded  to  la  a  yea  or  mo  faahion,  and  nay 
require  a  detailed  technical  explanation  and  ita  aaaociated 
metrica  to  reflect  attainment. 
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